Drag detection in main window is way too sensitive, causing some taps to be "ignored"

[svobs]

System and IINA version:

• macOS 13.1
• IINA 1.3.1 (latest develop: da1e06c)

I've been frustrated by this issue for quite a while now, and although I couldn't find anything via a quick search, I'm pretty sure I've come across at least a couple user reports that noted there was "no way to close the side panel" or that clicking to close didn't work reliably. I finally committed a big chunk of time to tracking it down, and I think I've found the culprit.

To get a better view of how clicks were being registered, I placed these logging statements at the start of these methods in MainWindowController:

  override func mouseDown(with event: NSEvent) {
    Logger.log("MainWindow mouseDown \(event.locationInWindow)", level: .verbose, subsystem: player.subsystem)
...

  override func mouseDragged(with event: NSEvent) {
    Logger.log("MainWindow mouseDrag \(event.locationInWindow)", level: .verbose, subsystem: player.subsystem)
...

  override func mouseUp(with event: NSEvent) {
  Logger.log("MainWindow mouseUp! isDragging: \(isDragging), isResizingSidebar: \(isResizingSidebar), clickCount: \(event.clickCount)", level: .verbose, subsystem: player.subsystem)
...

When I ran IINA with my test logging, I saw messages in the log like these:

01:06:22.638 [player0][v] MainWindow: mouseDown: (247.8671875, 417.46875)
01:06:22.640 [player0][v] MainWindow: mouseDragged: (247.8671875, 417.5859375)
01:06:22.767 [player0][v] MainWindow mouseUp! isDragging: true, isResizingSidebar: false, clickCount=1

This declared my tap was a "drag" (and thus not a click) even though my click lasted for 127ms and the drag distance was only 0.1pt in screen coordinates!

I should point out that I'm using the trackpad on a 16" MacBook Pro M1 Pro. Unlike most developers, I never use a mouse. (It's one of the main reasons I use Macs instead of other operating systems: multi-touch on on Linux/Windows has been almost unusable). And I happen to wear a glove size XL - I've got large hands and fingers. So I can see how this problem may not be noticeable for a mouse user, or someone who has smaller fingers. But for me, Apple's trackpad is so sensitive that it will send a "drag" notification even if my finger does a tiny rolling motion during a tap, or if my motion isn't straight down and up again. This is an unreasonable amount of precision to expect, and it doesn't align with other apps' behavior.

Expected behavior:

If the user does what a reasonable person considers to be a "tap", IINA should respond to it as a tap. It should only think the user is dragging once they drag some very small minimum distance, or hold it down longer than some reasonable time threshold, or both.

Actual behavior:

Sometimes users' taps or clicks are basically ignored because IINA incorrectly concluded that they wanted to quickly move the window by a single pixel or less.

Steps to reproduce:

See above.

How often does this happen?

When trying to close a side panel by tapping the video, there's about a 50-75% chance that my first tap will fail and a second attempt is needed. Probably the frequency varies widely between different users based on their usage patterns, and maybe trackpad users are more affected than mouse users.

[low-batt]

On whether IINA users are experiencing this issue. I believe this is the cause of issue #3198. The behavior reported in the last comment in issue ##3136 could also be due to this issue. So yes.

And I believe I have encountered this. When there is a flat surface nearby I use a high end Logitech mouse. I prefer the lack of the rubber band effect when scrolling. Less often I will use the touchpad. I can remember quickly tapping and being surprised the side panel did not close. As it closed when I tried again I assumed the tap had not registered. I'm now thinking it was this issue.

However as the nice debug logging shows AppKit has interpreted the mouse actions as dragging, this seems like a general macOS issue and not an IINA issue. Searching the net I found it hard to locate posts regarding this issue, but I did find a few posts complaining about the behavior of macOS. So up first is to confirm macOS does not provide any settings to address this.

I believe the relevant Apple documentation is Change Pointer Control settings for accessibility on Mac. The Trackpad Options section lists some settings that control dragging. I didn't see anything that would address the issue at hand.

I'm surprised this is not addressed by a macOS Accessibility setting.

Next is to check to see what Apple's Human Interface Guidelines has to say about this. In the Pointing devices section under Best practices there is this:

Be consistent when responding to mouse and trackpad gestures. People expect most gestures to work the same throughout the system, regardless of the app or game they’re using.

Reading that section my interpretation is that the primary concern is about not changing the high level behavior of a standard gesture. The HIG does not get into a lower level issue like the one at hand. But the "work the same" statement could be interpreted as covering not altering how dragging works.

Is there is any precedence for this in IINA? In Settings under the Control tab there is Sensitivity for normal seek and Sensitivity for volume. My thoughts on this is that these are higher level controls and not the same as customizing low level behavior.

Is it a common application practice to establish its own threshold for the mouse to move before accepting the operating system's dragging interpretation? I found nothing in Apple's documentation and examples. I was also unable to find any macOS examples on the net. I found the Stack Overflow JavaScript post How to distinguish mouse "click" and "drag" from 10 years ago to which there are some replies indicating the need to "require a minimum delta" before interpreting a mouse movement as a drag.

I bounced this off another developer (not an IINA contributor). He had a very strong opinion that if a macOS application were to establish its own delta then it must provide a setting for the user to opt into this. By default the application must accept the interpretation of macOS.

Given all that, here are my thoughts…

• I'm really surprised that macOS does not provide an accessibility setting to control this behavior

• As IINA users have reported encountering this it is desirable for IINA to address this issue

• IINA should provide some sort of Sensitivity for dragging setting to allow users to opt into this behavior

Don't pay too much attention to my thoughts. I'm not a GUI programmer. Need to hear the opinions of experienced macOS GUI programmers.

[svobs]

After some more research and testing, couple of big points:

1. This only applies to a very specific case: when the user is clicking inside the player window, and not clicks on any of the controls, panels, or other windows. So there are only a few possibly things being done here: (a) the user's defined single-click action (if any), which, importantly, also closes the side panel, (b) double-click action on video, or titlebar, and (c) move the window. Probably (a) and (b) a lot more important than (c) - and we are tilting the balance in their favor.
2. I was mistaken about the behavior of the new code when dragging past the initial threshold. There is no "jump" of the window once it's decided that a drag has started. The code window?.performDrag(with: event) doesn't go back and look at the position when mouseDown was called. Instead, each mouseDragged event has delta values that only measure the distance since the previous drag event. So the result is that the the window starts to move normally after the threshold, as though the drag started from the moment after the threshold. The effect is so subtle, I was able to increase the threshold up to 10 pts before I just started to get some vague feeling that the window was "heavier" than normal windows. I had to watch it closely to notice when it started to move.

So, highly doubt the user will notice the effect on drag. I really don't think this is worth the effort of creating a new user setting for.

In fact, it seems like my 2 pts distance threshold is smaller than average - I might suggest increasing it to 5. Of the answers I found online, two gave 5, one gave 6, though I found one that did suggest 2. Just playing with several random apps (Finder icons, Chrome tabs, GMail messages...) - feels like 4-5 is pretty typical. Although almost all of them seem to jump when they hit the threshold... I'm not opposed to changing the code to add in the jump, though I am a bit fond of the smoothness of this solution, tbh ;)

I'm really surprised that macOS does not provide an accessibility setting to control this behavior

I think I'm beginning to understand a bit better why Apple might have left it out. Had there been one, users might have assumed that it should apply to all applications. I found this discussion which goes into some examples of how there isn't really a one-size-fits-all approach.

I prefer the lack of the rubber band effect when scrolling.

Try disabling Settings > Accessibility > Pointer Control > Use inertia when scrolling.

On that same screen, I suppose the incredibly misleading checkbox Use trackpad for dragging option below that is Apple's answer for accessible dragging - one can use two- or three-finger drag. Yet checked or unchecked, single-finger drag is still enabled...

[low-batt]

That Stack Exchange post is really interesting. Good points about the proposed change being isolated and not needing to be controllable by the user. I have to defer to the other IINA developers on this as my background is not in UIs.

You will find posts on the net recommending disabling Use inertia when scrolling in order to avoid the rubber band effect when scrolling. However the rubber band effect is still present when that macOS setting is disabled.

Similar deal with the Use trackpad for dragging. Found lots of responses on the net indicating that setting could be used to address this issue. As you indicated it does not do what you would think from the name of the setting.

[svobs]

I've always found it more difficult to argue why some frontend decision is better than others. Maybe because it's so wrapped up in aesthetics, and the fact that what is more usable is frequently not the same is what is more logical. And to some, it is just intuitive, while to others, not at all obvious. Often it seems to come down to the project owners and trusting that they make the right choices...

I pushed an update to the PR which increased the drag threshold to 4.0 points. I found at 2pts, when I made really sloppy clicks it would still register as a drag, and seems like at 4.0 it still doesn't hinder legitimate drags.

[low-batt]

This is fixed in the released version of IINA.