Web Port in iOS Safari doesn't register clicks

[ericoporto]

Describe the bug
When running our Web port in the iOS Safari browser, clicks are not registered

The reason it appears to be that Touch.identifier works differently in it. The 2013 spec states:

identifier of type long, readonly
An identification number for each touch point. When a touch point becomes active, it must be assigned an identifier that is distinct from any other active touch point. While the touch point remains active, all events that refer to it must assign it the same identifier.

Here is a log after I added three log functions in sys_events.cpp, this is in a simulator, so the mouse is only one finger, I click, move and release I think six times and get this.

[Log] 20:31:36:062: on_sdl_touch_down: fingerId 998779017, xy (0.611224, 0.570836), dxdy (0.000000, 0.000000); (tst, line 583)
[Log] 20:31:36:096: on_sdl_touch_motion: fingerId 998779017, xy (0.611224, 0.570243), dxdy (0.000000, -0.000593); (tst, line 583)
[Log] 20:31:36:151: on_sdl_touch_up: fingerId 998779017, xy (0.611224, 0.570243), dxdy (0.000000, 0.000000); (tst, line 583)
[Log] 20:31:39:831: on_sdl_touch_down: fingerId 998779018, xy (0.586735, 0.581506), dxdy (0.000000, 0.000000); (tst, line 583)
[Log] 20:31:39:895: on_sdl_touch_motion: fingerId 998779018, xy (0.586735, 0.580913), dxdy (0.000000, -0.000593); (tst, line 583)
[Log] 20:31:39:948: on_sdl_touch_up: fingerId 998779018, xy (0.586735, 0.580913), dxdy (0.000000, 0.000000); (tst, line 583)
[Log] 20:31:51:444: on_sdl_touch_down: fingerId 998779019, xy (0.518367, 0.548903), dxdy (0.000000, 0.000000); (tst, line 583)
[Log] 20:31:51:485: on_sdl_touch_motion: fingerId 998779019, xy (0.518367, 0.548311), dxdy (0.000000, -0.000593); (tst, line 583)
[Log] 20:31:51:485: on_sdl_touch_up: fingerId 998779019, xy (0.518367, 0.548311), dxdy (0.000000, 0.000000); (tst, line 583)
[Log] 20:32:49:277: on_sdl_touch_down: fingerId 998779020, xy (0.271429, 0.595732), dxdy (0.000000, 0.000000); (tst, line 583)
[Log] 20:32:49:278: on_sdl_touch_motion: fingerId 998779020, xy (0.271429, 0.594547), dxdy (0.000000, -0.001186); (tst, line 583)
[Log] 20:32:49:278: on_sdl_touch_motion: fingerId 998779020, xy (0.275510, 0.594547), dxdy (0.004082, 0.000000); (tst, line 583)
[Log] 20:32:49:321: on_sdl_touch_motion: fingerId 998779020, xy (0.303061, 0.594547), dxdy (0.027551, 0.000000); (tst, line 583)
[Log] 20:32:49:323: on_sdl_touch_motion: fingerId 998779020, xy (0.488776, 0.598103), dxdy (0.185714, 0.003557); (tst, line 583)
[Log] 20:32:49:389: on_sdl_touch_motion: fingerId 998779020, xy (0.518367, 0.597510), dxdy (0.029592, -0.000593); (tst, line 583)
[Log] 20:32:49:389: on_sdl_touch_motion: fingerId 998779020, xy (0.544898, 0.596918), dxdy (0.026531, -0.000593); (tst, line 583)
[Log] 20:32:49:439: on_sdl_touch_motion: fingerId 998779020, xy (0.550000, 0.596918), dxdy (0.005102, 0.000000); (tst, line 583)
[Log] 20:32:49:822: on_sdl_touch_up: fingerId 998779020, xy (0.550000, 0.596918), dxdy (0.000000, 0.000000); (tst, line 583)
[Log] 20:32:54:715: on_sdl_touch_down: fingerId 998779021, xy (0.225510, 0.544754), dxdy (0.000000, 0.000000); (tst, line 583)
[Log] 20:32:54:773: on_sdl_touch_motion: fingerId 998779021, xy (0.225510, 0.543568), dxdy (0.000000, -0.001186); (tst, line 583)
[Log] 20:32:54:871: on_sdl_touch_motion: fingerId 998779021, xy (0.225510, 0.542976), dxdy (0.000000, -0.000593); (tst, line 583)
[Log] 20:32:55:072: on_sdl_touch_motion: fingerId 998779021, xy (0.253061, 0.553646), dxdy (0.027551, 0.010670); (tst, line 583)
[Log] 20:32:55:073: on_sdl_touch_motion: fingerId 998779021, xy (0.277551, 0.561944), dxdy (0.024490, 0.008299); (tst, line 583)
[Log] 20:32:55:126: on_sdl_touch_motion: fingerId 998779021, xy (0.318367, 0.573800), dxdy (0.040816, 0.011855); (tst, line 583)
[Log] 20:32:55:126: on_sdl_touch_motion: fingerId 998779021, xy (0.371429, 0.587433), dxdy (0.053061, 0.013634); (tst, line 583)
[Log] 20:32:55:169: on_sdl_touch_motion: fingerId 998779021, xy (0.404082, 0.593954), dxdy (0.032653, 0.006520); (tst, line 583)
[Log] 20:32:55:170: on_sdl_touch_motion: fingerId 998779021, xy (0.441837, 0.602845), dxdy (0.037755, 0.008892); (tst, line 583)
[Log] 20:32:55:253: on_sdl_touch_motion: fingerId 998779021, xy (0.468367, 0.606995), dxdy (0.026531, 0.004149); (tst, line 583)
[Log] 20:32:55:254: on_sdl_touch_motion: fingerId 998779021, xy (0.512245, 0.611144), dxdy (0.043878, 0.004149); (tst, line 583)
[Log] 20:32:55:307: on_sdl_touch_motion: fingerId 998779021, xy (0.525510, 0.611737), dxdy (0.013265, 0.000593); (tst, line 583)
[Log] 20:32:55:307: on_sdl_touch_motion: fingerId 998779021, xy (0.534694, 0.611737), dxdy (0.009184, 0.000000); (tst, line 583)
[Log] 20:32:55:580: on_sdl_touch_up: fingerId 998779021, xy (0.534694, 0.611737), dxdy (0.000000, 0.000000); (tst, line 583)

AGS Version
it happens on current 3.6.0.45, but curiously it did not happen in an old version.

Game
https://ericoporto.github.io/agsjs/tst/

To Reproduce
Steps to reproduce the behavior:

1. Load the URL and see that touching the screen does nothing.
2. If you have Safari Technology Preview installed on a macOS you can connect to the iOS system with Safari
3. the log will have the mentioned on_sdl_touch log messages

Expected behavior
It's expected to work

Smartphone:

• Device: iOS Simulator with iPhone 14 Pro Max
• OS: iOS
• Version 16.2

Additional context
When SDL own touch emulation was used, there was no reliance in fingerId, so it worked with that.

The API from SDL only states that fingerId is unique, and nothing else, so apparently we can't rely on actual values from it - even though on Android it's consistent with both the App and the Web version on different browsers, it appears on other devices this is not guaranteed.

[ivan-mogilko]

I guess it should be same between touch down and touch up, but may be a different one next time it's down?
We may save which finger IDs were down in variable(s) to be able to count number of fingers, and which are moved/upped.
EDIT: and do not expect finger IDs to go 0,1,2,3... but be anything.

[ericoporto]

I guess it should be same between touch down and touch up, but may be a different one next time it's down?

Yeah, on my log above they are consistent for a down-motion-up sequence and then next sequence they get a new number.

and do not expect finger IDs to go 0,1,2,3... but be anything.

In this case they appear to be consecutive but I get a feeling they use all the range of the 64-bit id they have. But yeah, I think the idea is to work only with the assumption that each finger ID is unique for all the finger IDs "active" (simultaneous?) - on screen or just leaving the screen.

In the mouse emulation, for the two finger method, can't quite figure the case where one finger is on screen and then a second one is used (for the right click) and then with the second one on screen, the first one is lifted. My guess is we just don't support this, the second finger should only be used for tapping.

https://wiki.libsdl.org/SDL2/SDL_TouchFingerEvent

I noticed that there's also a timestamp member that could be used if needed.

[ivan-mogilko]

Well, following are uses of fingerID:

1. detect_double_tap() function. Seem to work fine with random ID, as it records them in a variable already.
2. touch.fingers_down and tfinger_to_mouse_but(). First is a bitfield, the second is function. But I put them together as same problem, because both basically convert from fingerID, assuming that it's 0-based and each new finger takes the free ID from the start (i.e. if fingerID 0 is free, then the next finger will be 0).

In regards to the p2, the issue here is that in the current logic, the pressed fingers ares kind of "locked" to certain role. As you mentioned above:

In the mouse emulation, for the two finger method, can't quite figure the case where one finger is on screen and then a second one is used (for the right click) and then with the second one on screen, the first one is lifted. My guess is we just don't support this, the second finger should only be used for tapping.

Perhaps the solution may be to remember the finger's role until all of them are released. That is, if you:

• first finger down (consider LMB);
• second finger down (consider RMB);
• first finger up (consider LMB up);
• second finger is still pressed and still considered RMB;
• third(?) finger down, takes the free role of LMB.

And alternate solution is to rely solely on number of fingers. Two examples:
ex 1

• first finger down;
• first same finger up -- that's a LMB click.

ex 2

• first finger down;
• second finger down;
• any finger released -- that's a RMB click.
• any new finger down.
• any finger up -- that's again a RMB click.

[ericoporto]

detect_double_tap() function. Seem to work fine with random ID, as it records them in a variable already.

Yes, this is true, the minor note is last_tap_finger can't be an int, it needs to be 64-bit or SDL_FingerID type.

Perhaps the solution may be to remember the finger's role until all of them are released.

This idea of remembering roles looks more correct - it kinda emulates what Android is doing, like first finger down it reserves the 0 ID, and then second finger is down it reserves the 1 ID, if first finger is released and 0 ID is released and then third finger is down 0 ID is again reserved.

This is something that if existed could be reused in a future in AGS touch events, as this would give fingerIDs consistent between platforms. I just wonder if there's something really hard about this, since it sounds like it should be in SDL2 itself, but an alternative reason why it isn't may just be because most of the sdl2 people care primarily for PC gaming.

And alternate solution is to rely solely on number of fingers

This reads like it is easier to implement.

[ericoporto]

Made a fix that works apparently, not sure how good it is: c9b3152 (it's the second alternative, that simply uses number of fingers)

Edit: went and attempted the first idea here master...ericoporto:ags:fix-web-ios-touch Edit: made a PR from this.

Can be tested here: https://ericoporto.github.io/agsjs/tst2/