NEW: Added achievable frequency and latency (processing delay) to Input Debugger (ISX-2254) #2142

ekcoh · 2025-02-19T15:27:20Z

Description

Adds achievable sample/event frequency (as well as sensor frequency and global polling frequency) to Input Debugger device window. These are just basic calculations over periods of one second at the moment similar to FPS counter.
Adds input processing delay (average, min, max) to Input Debugger device window.

The diagnostic values are similar to FPS updated once a second to show primarily the average. This is intended to help with diagnosing the behaviour of the system and devices and help spot problems.

Added 11 new test cases to cover the calculators as part of editor test assembly which required modification to accept unsafe code in the test assembly which I believe should be ok to change.

Target frequency shown is based on IOCTL sample frequency query response which I believe is only supported by sensors currently. Polling frequency is as we know global at the moment, but showing it since it relates to devices with polling backends which we currently cannot determine from the system. If both of these could be capability queried in the future we can make a better message here.

Testing status & QA

Tested manually with mouse, keyboard and gamepad on macOS during development.
Added unit tests to verify that calculations are working as expected.

Overall Product Risks

Very small. Has no impact on production or player code.

Complexity: 0
Halo Effect: 0

Comments to reviewers

If possible to cover sensors in review/QA it would be great.

Note that both label fields have tooltip that also needs review.

Checklist

Before review:

Changelog entry added.
- Explains the change in Changed, Fixed, Added sections.
- For API change contains an example snippet and/or migration example.
- JIRA ticket linked, example (case %%). If it is a private issue, just add the case ID without a link.
- Jira port for the next release set as "Resolved".
Tests added/changed, if applicable.
- Functional tests: 11 new test cases added to InputLatencyCalculatorTests and SampleFrequencyCalculatorTests.
- Performance tests.
- Integration tests.
Docs for new/changed API's.
- Xmldoc cross references are set correctly.
- Added explanation how the API works.
- Usage code examples added.
- The manual is updated, if needed.

During merge:

Commit message for squash-merge is prefixed with one of the list:
- NEW: ___.
- FIX: ___.
- DOCS: ___.
- CHANGE: ___.
- RELEASE: 1.1.0-preview.3.

After merge:

Create forward/backward port if needed. If you are blocked from creating a forward port now please add a task to ISX-1444.

…ebugger

…thub.com:Unity-Technologies/InputSystem into isx-2254-frequency-and-latency-in-input-debugger

ritamerkl

It's a great to have, thanks for adding this!
I only did a quick run through but it looks good from my POV. I need to take another closer look and try it myself.

Packages/com.unity.inputsystem/InputSystem/Editor/Debugger/InputDeviceDebuggerWindow.cs

ekcoh · 2025-02-21T12:01:01Z

Packages/com.unity.inputsystem/InputSystem/Editor/Debugger/SampleFrequencyCalculator.cs

+        private double m_LastUpdateTime;
+        private int m_SampleCount;
+
+        public SampleFrequencyCalculator(float targetFrequency, double realtimeSinceStartup)


Target frequency is currently only a "convient storage" here. Hower I would suggest keeping it inside this type since an evolved implementation will likely use it to compute error compared to target and create a CDF or PDF of it to illustrate the applications or architectures ability to promptly react to stimuli.

…ebugger

surfnerd · 2025-02-25T19:33:35Z

I think this is great step toward transparency of processing of input events. I think it should be made clear that this doesn't measure the latency between the creation of an input event to when it shows up on the screen, but from the creation of the event to when it's processed by the input system.

ekcoh · 2025-02-25T19:45:11Z

I think this is great step toward transparency of processing of input events. I think it should be made clear that this doesn't measure the latency between the creation of an input event to when it shows up on the screen, but from the creation of the event to when it's processed by the input system.

Agreed, that's a much simpler/better way to say what I am trying to say in that tool tip. Also why I renamed the label to "Processing Delay". Will try to improve the tooltip based your description here. Thanks! :)

surfnerd · 2025-02-25T20:37:13Z

Perhaps the manual could be updated with the new screenshot, and a little blurb about what value this adds could be helpful as well.

…Debugger screenshot.

Pauliusd01 · 2025-02-27T07:49:38Z

Is it expected that I'm seeing processing delay this high when plugging in an android device? Is it because It's a "remotely" connected one?

Pauliusd01 · 2025-02-27T09:43:16Z

Also, the duelsense controllers have a really high sample frequency, but that might be due to the them being very noisy?

ekcoh · 2025-02-27T09:58:41Z

Is it expected that I'm seeing processing delay this high when plugging in an android device? Is it because It's a "remotely" connected one?

You probably found another issue with remote devices having some unusable timestamps. This is probably a root of other bad things to happen. Great find. Did you use Unity remote to get this happening?

I decided to cap the "age" at 1000.0 ms and this is what you see. However this is only based on current time - event timestamp. If event timestamp comes from another clock (remote) device, this doesn't make sense, so it probably means remote devices are not re-timestamped when coming in to the host or the remote protocol do not allow sending the remote device time to allow computing the true latency (including network latency). I think its unrelated to the PR, but we might to either filter remote ones out for now until we have figured that part out or do something similar.... I didn't try remote when doing this.

Pauliusd01 · 2025-02-27T10:18:52Z

Did you use Unity remote to get this happening?

The steps to repro are: Connect an android device and build with the development build type -> In the Input Debugger click on the Remote Devices dropdown and pick your android device -> in the android devices list double click any device -> Processing Delay is always above 1k ms

ekcoh · 2025-02-27T11:56:06Z

Also, the duelsense controllers have a really high sample frequency, but that might be due to the them being very noisy?

Not really, on Windows which was the platform you tested on, DualSense is processed over HID. The current HID implementation do not have any rate-adoption (apart from max read requests in flight I believe) so it will run as fast as the underlying HID driver allows it to do. It's a good observation since this shows processing differences in platform-dependent code that could be up for discussion. However, you are correct that the noise is part of it as well since the noise will allow events/samples to be forwarded since the state "changes". If there was no noise or if signal was pre filtered it would reduce the frequency while the device is idle.

ekcoh · 2025-02-27T11:56:54Z

E.g. on macOS where I am currently working with another DualSense bug, the achieved frequency is around 65 Hz. So 1/8 compared to Win at the moment.

Pauliusd01

Checked debugger window for these devices: touch, keyboard, mouse, PS and Xbox controllers and commonly used android sensors.

ekcoh · 2025-02-27T12:38:32Z

Thanks for the rigorous testing of this debug feature @Pauliusd01

ritamerkl

Approving to unblock this, will add more thoughts next week !

ritamerkl · 2025-03-03T13:01:49Z

I tested the newly added Input Debugger features myself now, and think it is priceless. The only wish I would have is to name target/ Polling Frequency with a bit more clarity or adding an explanation with a small text underneath, or a hover textbox? For me the meaning wasn't clear initially (your live demo on friday was very helpful) - also for the restrictions eg when n/a shows up and what this means.

ekcoh · 2025-03-04T08:49:44Z

I tested the newly added Input Debugger features myself now, and think it is priceless. The only wish I would have is to name target/ Polling Frequency with a bit more clarity or adding an explanation with a small text underneath, or a hover textbox? For me the meaning wasn't clear initially (your live demo on friday was very helpful) - also for the restrictions eg when n/a shows up and what this means.

I agree, the UI layout isn't very helpful, maybe I should try revisit the IMGUI design here to at least make it a bit better (I think the debugger window is overpopulated to begin with, contains a mix of static and dynamic data and should likely be split into multiple windows and/or perspectives).

Regarding the N/A and Target polling rate I would propose the following:

Add a new IOCTL command allowing to query whether a device is subject for polling or not. If it's not, there is no point in showing Polling Frequency. This would be trivial if done on module level and could even be a flag. Will investigate.
Only show target if the device responds that it has a device-specific target frequency. This will only show something for sensors currently. In the future I hope all devices can be device-specific.

Taking both above into account maybe we should do this before landing this:

Dynamically change "Sampling Frequency" label into "Sample Frequency" if it's a polling device and "Event Frequency" if it's not a polling device. Only for Sampling Frequency would we show a "Target" and the target could mean device-specific target or global polling frequency setting dependencies on device. Then the info is more contextual.

What do you think?

Adding the IOCTL could be done on this branch in managed and separately in native. It just needs a good fallback when we cannot tell. When we cannot tell, maybe we should just hide Target since it doesn't add any info anyways.

ritamerkl · 2025-03-10T10:48:39Z

I tested the newly added Input Debugger features myself now, and think it is priceless. The only wish I would have is to name target/ Polling Frequency with a bit more clarity or adding an explanation with a small text underneath, or a hover textbox? For me the meaning wasn't clear initially (your live demo on friday was very helpful) - also for the restrictions eg when n/a shows up and what this means.

I agree, the UI layout isn't very helpful, maybe I should try revisit the IMGUI design here to at least make it a bit better (I think the debugger window is overpopulated to begin with, contains a mix of static and dynamic data and should likely be split into multiple windows and/or perspectives).

Regarding the N/A and Target polling rate I would propose the following:

Add a new IOCTL command allowing to query whether a device is subject for polling or not. If it's not, there is no point in showing Polling Frequency. This would be trivial if done on module level and could even be a flag. Will investigate.

Only show target if the device responds that it has a device-specific target frequency. This will only show something for sensors currently. In the future I hope all devices can be device-specific.

Taking both above into account maybe we should do this before landing this:

Dynamically change "Sampling Frequency" label into "Sample Frequency" if it's a polling device and "Event Frequency" if it's not a polling device. Only for Sampling Frequency would we show a "Target" and the target could mean device-specific target or global polling frequency setting dependencies on device. Then the info is more contextual.

What do you think?

Adding the IOCTL could be done on this branch in managed and separately in native. It just needs a good fallback when we cannot tell. When we cannot tell, maybe we should just hide Target since it doesn't add any info anyways.

Sounds good to me!

chrstphrsxtn

Reviewing on behalf of docs. Approving so as not to block, but left some suggestions which should be applied before merging.

Packages/com.unity.inputsystem/Documentation~/Debugging.md

chrstphrsxtn · 2025-03-12T12:17:18Z

Packages/com.unity.inputsystem/Documentation~/Debugging.md

@@ -42,7 +42,11 @@ In the Input Debugger window, navigate to the __Devices__ list and double-click

 ![Device in Input Debugger](Images/DeviceInDebugger.png)

-The top of the Device window displays general information about the specific Device, such as name, manufacturer, and serial number.
+The top of the Device window displays general information about the specific Device, such as name, manufacturer, associated layout, device flags, device ID and serial number.  In addition, this section also display the current __sample frequency__ and __processing delay__ of the deivce.


Suggested change

The top of the Device window displays general information about the specific Device, such as name, manufacturer, associated layout, device flags, device ID and serial number. In addition, this section also display the current __sample frequency__ and __processing delay__ of the deivce.

The top of the Device window displays general information about the specific Device, such as name, manufacturer, associated layout, device flags, device ID and serial number. This section also displays the current __sample frequency__ and __processing delay__ of the device.

chrstphrsxtn · 2025-03-12T12:19:22Z

Packages/com.unity.inputsystem/Documentation~/Debugging.md

+
+__Sample frequency__ indicates the current frequency of samples or events the Input System is currently processing at given in Hertz (Hz). For devices receiving events this reflects the current flow of events received by the system, and for device receiving periodic readings this reflects the achievable sample rate of the system. The latter may be compared to the globally configured target sampling frequency, while achievable event frequency is uncorrelated to the sample polling frequency setting.
+
+__Processing delay__ indicates the average, minimum and maximum latency contribution from creating an input event or reading until the Input System has processed the same input event. Note that this excludes any additional input delay caused by OS, drivers or device communication. Also note that this excludes any additional output latency that may be caused by additional processing, rendering, GPU swap-chains or display refresh rate.


Suggested change

__Processing delay__ indicates the average, minimum and maximum latency contribution from creating an input event or reading until the Input System has processed the same input event. Note that this excludes any additional input delay caused by OS, drivers or device communication. Also note that this excludes any additional output latency that may be caused by additional processing, rendering, GPU swap-chains or display refresh rate.

__Processing delay__ indicates the average, minimum, and maximum time delay from when an input event or reading is created until the Input System has processed it. Note that this excludes any additional input delay caused by OS, drivers, or device communication. Also note that this excludes any additional output latency that may be caused by additional processing, rendering, GPU swap-chains, or display refresh rate.

chrstphrsxtn · 2025-03-12T13:01:50Z

Packages/com.unity.inputsystem/InputSystem/Editor/Debugger/InputDeviceDebuggerWindow.cs

+                EditorGUILayout.LabelField(new GUIContent("Sample Frequency", sampleFrequencyTooltip), new GUIContent(m_DeviceFrequencyString), EditorStyles.label);
+            const string processingDelayTooltip =
+                "Displays the average, minimum and maximum observed input processing delay. This shows the time from " +
+                "when an input event is first created within Unity until its processed by the Input System. " +


Suggested change

"when an input event is first created within Unity until its processed by the Input System. " +

"when an input event is first created within Unity until it's processed by the Input System. " +

Packages/com.unity.inputsystem/InputSystem/Editor/Debugger/InputDeviceDebuggerWindow.cs

Improved documentation Co-authored-by: chrstphrsxtn <74899224+chrstphrsxtn@users.noreply.github.com>

…utDeviceDebuggerWindow.cs Co-authored-by: chrstphrsxtn <74899224+chrstphrsxtn@users.noreply.github.com>

…ebugger

ekcoh added 2 commits February 19, 2025 16:21

ADD: Added frequency and latency (processing delay) to Input debugger.

7cb1b4c

Updated changelog.

0a77be5

ekcoh requested review from jfreire-unity, ritamerkl and Pauliusd01 February 19, 2025 15:27

ekcoh changed the title ~~ADD: Added achievable frequency and latency (processing delay) to Input Debugger (ISX-2254)~~ ISX-2254 NEW: Added achievable frequency and latency (processing delay) to Input Debugger Feb 19, 2025

ekcoh added 3 commits February 19, 2025 16:33

Merge branch 'develop' into isx-2254-frequency-and-latency-in-input-d…

cdf7ad1

…ebugger

Fixed formatting

0a9a9ad

Merge branch 'isx-2254-frequency-and-latency-in-input-debugger' of gi…

d479d44

…thub.com:Unity-Technologies/InputSystem into isx-2254-frequency-and-latency-in-input-debugger

ekcoh changed the title ~~ISX-2254 NEW: Added achievable frequency and latency (processing delay) to Input Debugger~~ NEW: Added achievable frequency and latency (processing delay) to Input Debugger (ISX-2254) Feb 19, 2025

ekcoh added 2 commits February 20, 2025 10:43

Added unit tests for debugger calculators.

45573a3

Updated tooltip

37b18db

ekcoh requested review from surfnerd and unity-andrewc February 20, 2025 11:07

ritamerkl reviewed Feb 21, 2025

View reviewed changes

Packages/com.unity.inputsystem/InputSystem/Editor/Debugger/InputDeviceDebuggerWindow.cs Outdated Show resolved Hide resolved

ekcoh commented Feb 21, 2025

View reviewed changes

Packages/com.unity.inputsystem/InputSystem/Editor/Debugger/InputDeviceDebuggerWindow.cs Outdated Show resolved Hide resolved

ekcoh commented Feb 21, 2025

View reviewed changes

ekcoh added 2 commits February 21, 2025 16:03

Removed partial keyword accidentally added during refactoring

40948db

Updated tooltip and field label based on review discussion

8af0ad1

ekcoh requested a review from duckets February 21, 2025 15:09

ekcoh added 2 commits February 24, 2025 13:56

Merge branch 'develop' into isx-2254-frequency-and-latency-in-input-d…

3cfa59d

…ebugger

Merge branch 'develop' into isx-2254-frequency-and-latency-in-input-d…

8175826

…ebugger

surfnerd approved these changes Feb 25, 2025

View reviewed changes

Added info to manual for new Input Debugger metrics, updated DeviceIn…

30c285a

…Debugger screenshot.

Pauliusd01 approved these changes Feb 27, 2025

View reviewed changes

ritamerkl approved these changes Feb 28, 2025

View reviewed changes

chrstphrsxtn approved these changes Mar 12, 2025

View reviewed changes

chrstphrsxtn reviewed Mar 12, 2025

View reviewed changes

Packages/com.unity.inputsystem/InputSystem/Editor/Debugger/InputDeviceDebuggerWindow.cs Outdated Show resolved Hide resolved

ekcoh and others added 2 commits March 13, 2025 13:12

Update Packages/com.unity.inputsystem/Documentation~/Debugging.md

2858da1

Improved documentation Co-authored-by: chrstphrsxtn <74899224+chrstphrsxtn@users.noreply.github.com>

Update Packages/com.unity.inputsystem/InputSystem/Editor/Debugger/Inp…

ced5e1a

…utDeviceDebuggerWindow.cs Co-authored-by: chrstphrsxtn <74899224+chrstphrsxtn@users.noreply.github.com>

ekcoh removed request for duckets, jfreire-unity and unity-andrewc March 13, 2025 12:13

Merge branch 'develop' into isx-2254-frequency-and-latency-in-input-d…

24378e2

…ebugger

ekcoh merged commit ca0d36f into develop Mar 13, 2025
107 of 110 checks passed

ekcoh deleted the isx-2254-frequency-and-latency-in-input-debugger branch March 13, 2025 16:01

	The top of the Device window displays general information about the specific Device, such as name, manufacturer, associated layout, device flags, device ID and serial number. In addition, this section also display the current __sample frequency__ and __processing delay__ of the deivce.
	The top of the Device window displays general information about the specific Device, such as name, manufacturer, associated layout, device flags, device ID and serial number. This section also displays the current __sample frequency__ and __processing delay__ of the device.


		__Sample frequency__ indicates the current frequency of samples or events the Input System is currently processing at given in Hertz (Hz). For devices receiving events this reflects the current flow of events received by the system, and for device receiving periodic readings this reflects the achievable sample rate of the system. The latter may be compared to the globally configured target sampling frequency, while achievable event frequency is uncorrelated to the sample polling frequency setting.

		__Processing delay__ indicates the average, minimum and maximum latency contribution from creating an input event or reading until the Input System has processed the same input event. Note that this excludes any additional input delay caused by OS, drivers or device communication. Also note that this excludes any additional output latency that may be caused by additional processing, rendering, GPU swap-chains or display refresh rate.

	"when an input event is first created within Unity until its processed by the Input System. " +
	"when an input event is first created within Unity until it's processed by the Input System. " +

NEW: Added achievable frequency and latency (processing delay) to Input Debugger (ISX-2254) #2142

NEW: Added achievable frequency and latency (processing delay) to Input Debugger (ISX-2254) #2142

Uh oh!

Conversation

ekcoh commented Feb 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing status & QA

Overall Product Risks

Comments to reviewers

Checklist

Uh oh!

ritamerkl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ekcoh Feb 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

surfnerd commented Feb 25, 2025

Uh oh!

ekcoh commented Feb 25, 2025

Uh oh!

surfnerd commented Feb 25, 2025

Uh oh!

Pauliusd01 commented Feb 27, 2025

Uh oh!

Pauliusd01 commented Feb 27, 2025

Uh oh!

ekcoh commented Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pauliusd01 commented Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ekcoh commented Feb 27, 2025

Uh oh!

ekcoh commented Feb 27, 2025

Uh oh!

Pauliusd01 left a comment

Choose a reason for hiding this comment

Uh oh!

ekcoh commented Feb 27, 2025

Uh oh!

ritamerkl left a comment

Choose a reason for hiding this comment

Uh oh!

ritamerkl commented Mar 3, 2025

Uh oh!

ekcoh commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ritamerkl commented Mar 10, 2025

Uh oh!

chrstphrsxtn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chrstphrsxtn Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

chrstphrsxtn Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

chrstphrsxtn Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ekcoh commented Feb 19, 2025 •

edited

Loading

ekcoh Feb 21, 2025 •

edited

Loading

ekcoh commented Feb 27, 2025 •

edited

Loading

Pauliusd01 commented Feb 27, 2025 •

edited

Loading

ekcoh commented Mar 4, 2025 •

edited

Loading