zforce_ts / cyttsp5 touchscreen troubles (Kobo / Tolino)

[akemnade]

• KOReader version: 2022.08 (armhf debian package)
• Device: Kobo Clara HD (cyttsp5) and Tolino Shine 2HD (zforce_ts)

Environment

• Debian Bullseye with Xorg + nodm + drm modesetting driver
• near-mainline kernel (like in pmOS) 5.17.9, zforce_ts is unmodified mainline driver.
• Touchscreen set up so that it works well with other applications, and X cursor properly follows it

Issue

Touchscreen does not interact properly with KOReader

• wild tipping in the middle of the screen produces some screenshot dialogs
• top/bottom menus are not reachable
• home symbol can be clicked
• workaround: position the mouse cursor at the destination and execute DISPLAY=:0 xte "mouseclick 1" (e.g. from a ssh session)

Steps to reproduce

start koreader in environment described above and try to interact with it.

[NiLuJe]

That'd be SDL, so I'm curious as to what it's actually outputting. I only have a vague experience with it, do we have a way to dump whatever it's sending right now?

I don't remember if anyone's ever worked on devices with an actual touchscreen and SDL, does that ring a bell for anyone?

(Also, I don't think we have OTAs for the Debian builds, and I happen to have landed a fair bit of refactoring to gesture detection since the release, so I would discourage testing on the release ;)).

[Frenzie]

I don't remember if anyone's ever worked on devices with an actual touchscreen and SDL

Do you mean devices as in things like the Kobo Clara HD? 'cause the SDL stuff works great on a touchscreen, also multitouch (to be precise, the events are picked up by SDL; I haven't worked on forwarding them to front).

(Also, I don't think we have OTAs for the Debian builds, and I happen to have landed a fair bit of refactoring to gesture detection since the release, so I would discourage testing on the release ;)).

Not on ota.koreader.rocks but you can grab 'em from Gitlab.

[akemnade]

should I try to record someting with evemu-device/evemu-record? osk-sdl in test mode works at the same setup.

https://github.com/koreader/koreader/actions/workflows/build.yml?query=branch%3Amaster seems to produce only macOS stuff? So building master myself would be a starting point?

[pazos]

I don't remember if anyone's ever worked on devices with an actual touchscreen and SDL, does that ring a bell for anyone?

Oh yeah!

Works fine here:

• Ubuntu 22.04 (gnome) - Hp chromebook x360 (with UEFI firmware) in both X11 and wayland

• ChromeOS tablet - lenovo duet, running under crostini.

I also tried a couple of years ago a RPi with a touchscreen, running without X. SDL worked as is in the framebuffer (w/EGL driver) but failed to register input events. Didn't play much with the thing but the ts was MT-B and I think most of the code we already have can be repurposed for a device like that that uses generic linux events as input and sdl as output.

Back to OP issue, it could be something with the implementation of the ts drivers. Maybe it can be reproduced on other sdl apps?

[akemnade]

I tried osk-sdl in test mode on top of X and it gets its events. SDL here seems to get its events through X.
ssh -X kobo and starting koreader in that session also works.

[NiLuJe]

https://github.com/koreader/koreader/actions/workflows/build.yml?query=branch%3Amaster seems to produce only macOS stuff?

Yeah, only macOS is built on GitHub, the rest is built over on GitLab, you should find the debian stuff in the build artifacts.

[NiLuJe]

I have no experience with SDL, but, FWIW, the SDL -> evdev mapping happens here. At a very quick glance, it won't do multi-touch (and I have no idea how SDL even reports that ;p).

While fixing it would be nice, the proper solution (here, but especially for the fbdev kernels) would probably be a custom platform àla prstux that properly pokes at the evdev stuff directly and only uses SDL/fbdev for output like @pazos said. Depending on how much stuff needs to be implemented it might make sense to make it a subclass of the sdl/kobo platforms.

[Frenzie]

At a very quick glance, it won't do multi-touch

Sure it will, just handle the finger stuff instead of pretending it's a mouse click. As I said above I might do that someday if I were slightly bored, but that's a somewhat unlikely proposition…

[NiLuJe]

Oh, yeah, I meant "as-is" ^^.

[akemnade]

https://github.com/koreader/koreader/actions/workflows/build.yml?query=branch%3Amaster seems to produce only macOS stuff?

Yeah, only macOS is built on GitHub, the rest is built over on GitLab, you should find the debian stuff in the build artifacts.

well, it causes some library conflicts between mesa (oough, I hope it is not used) and libstdc++, so I cannot easily test that one. But I did some research using the last release

[akemnade]

touchfix.txt
that does the trick, It is on top of the last release (therefore no PR or something) these finger events seem to get zero values. No idea why, perhaps something is going mad with floats? I get both mousemotion and fingermotion.
It helps with both drivers.

[NiLuJe]

https://github.com/koreader/koreader/actions/workflows/build.yml?query=branch%3Amaster seems to produce only macOS stuff?

Yeah, only macOS is built on GitHub, the rest is built over on GitLab, you should find the debian stuff in the build artifacts.

well, it causes some library conflicts between mesa (oough, I hope it is not used) and libstdc++, so I cannot easily test that one. But I did some research using the last release

What's the exact error, for future reference?

(I'm assuming SDL is responsible for pulling in mesa for OpenGL output, FWIW).

EDIT: It doesn't do so directly on my desktop, so there may be hope there...

EDIT²: I don't recall if the debian builds might not also be shipping some libs from the build env as a poor man's workaround, which might not help the situation (ping @pazos).

[akemnade]

I have no experience with SDL, but, FWIW, the SDL -> evdev mapping happens here. At a very quick glance, it won't do multi-touch (and I have no idea how SDL even reports that ;p).

While fixing it would be nice, the proper solution (here, but especially for the fbdev kernels) would probably be a custom platform àla prstux that properly pokes at the evdev stuff directly and only uses SDL/fbdev for output like @pazos said. Depending on how much stuff needs to be implemented it might make sense to make it a subclass of the sdl/kobo platforms.

depends on the use-case If running on top of X11/weston amoung other apps, it is probably a bad idea to use evdev directly, because then it is assumed to handle input just like any other application and use whatever is configured.

If it is running more in a standalone way, things might be different.

[NiLuJe]

touchfix.txt that does the trick, It is on top of the last release (therefore no PR or something) these finger events seem to get zero values. No idea why, perhaps something is going mad with floats? I get both mousemotion and fingermotion. It helps with both drivers.

I'd need to double-check because I'm unfamiliar with the SDL_Event giant sack of union'ed structs, but that indeed looks like a bug (e.g., accessing the wrong layout in the union).

[akemnade]

https://github.com/koreader/koreader/actions/workflows/build.yml?query=branch%3Amaster seems to produce only macOS stuff?

Yeah, only macOS is built on GitHub, the rest is built over on GitLab, you should find the debian stuff in the build artifacts.

well, it causes some library conflicts between mesa (oough, I hope it is not used) and libstdc++, so I cannot easily test that one. But I did some research using the last release

What's the exact error, for future reference?

(I'm assuming SDL is responsible for pulling in mesa for OpenGL output, FWIW).

EDIT: It doesn't do so directly on my desktop, so there may be hope there...

EDIT²: I don't recall if the debian builds might not also be shipping some libs from the build env as a poor man's workaround, which might not help the situation (ping @pazos).

it includes some libraries, especially the libstdc++, creating mess here

andi@debnfs:~$ DISPLAY=:0 koreader &
[1] 14216
andi@debnfs:~$ /usr/bin/koreader: line 2: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
---------------------------------------------
                launching...
  _  _____  ____                _
 | |/ / _ \|  _ \ ___  __ _  __| | ___ _ __
 | ' / | | | |_) / _ \/ _` |/ _` |/ _ \ '__|
 | . \ |_| |  _ <  __/ (_| | (_| |  __/ |
 |_|\_\___/|_| \_\___|\__,_|\__,_|\___|_|

 It's a scroll... It's a codex... It's KOReader!

 [*] Current time: 09/11/22-21:00:27
 [*] Version: v2022.08-12-g6f5c229c_2022-09-10

ffi.load: blitbuffer
ffi.load: SDL2
ffi.load: SDL2
Starting SDL in /usr/lib/koreader/
libGL error: MESA-LOADER: failed to open swrast: /usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/arm-linux-gnueabihf/dri:\$${ORIGIN}/dri:/usr/lib/dri)
libGL error: failed to load driver: swrast
X Error of failed request:  GLXBadContext
  Major opcode of failed request:  151 (GLX)
  Minor opcode of failed request:  6 (X_GLXIsDirect)
  Serial number of failed request:  100
  Current serial number in output stream:  99

[1]+  Exit 1                  DISPLAY=:0 koreader

without the c**p at the beginning of the path, the following open should be successful

[pid 14756] openat(AT_FDCWD, "\\$/lib/arm-linux-gnueabihf/dri/swrast_dri.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)

Doing

root@debnfs:/usr/lib# ln -s arm-linux-gnueabihf/dri .

And starting again:

andi@debnfs:~$ DISPLAY=:0 koreader &
[1] 15084
andi@debnfs:~$ /usr/bin/koreader: line 2: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
---------------------------------------------
                launching...
  _  _____  ____                _
 | |/ / _ \|  _ \ ___  __ _  __| | ___ _ __
 | ' / | | | |_) / _ \/ _` |/ _` |/ _ \ '__|
 | . \ |_| |  _ <  __/ (_| | (_| |  __/ |
 |_|\_\___/|_| \_\___|\__,_|\__,_|\___|_|

 It's a scroll... It's a codex... It's KOReader!

 [*] Current time: 09/11/22-21:06:37
 [*] Version: v2022.08-12-g6f5c229c_2022-09-10

ffi.load: blitbuffer
ffi.load: SDL2
ffi.load: SDL2
Starting SDL in /usr/lib/koreader/

andi@debnfs:~$ libGL error: MESA-LOADER: failed to open swrast: /usr/bin/../lib/koreader/libs/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /lib/arm-linux-gnueabihf/libLLVM-11.so.1) (search paths /usr/lib/arm-linux-gnueabihf/dri:\$${ORIGIN}/dri:/usr/lib/dri)
libGL error: failed to load driver: swrast
X Error of failed request:  GLXBadContext
  Major opcode of failed request:  151 (GLX)
  Minor opcode of failed request:  6 (X_GLXIsDirect)
  Serial number of failed request:  100
  Current serial number in output stream:  99

[NiLuJe]

I get both mousemotion and fingermotion.

That's... interesting. And also rings a bell? (#9091).

It's possible the lack of MT support in the code flags the wrong stuff as down/up and as such doesn't generate contatc moves or... something?

[akemnade]

generated events out of fingermotion have zero x/y, probably something rounding down the float ranging from 0..1) to zero.

[NiLuJe]

Re: #9506 (comment)

I can't comment on the funky LD_LIBRARY_PATH (ping @pazos), but shipping the STL in Debian builds is definitely an oversight on my part ;).

[NiLuJe]

generated events out of fingermotion have zero x/y, probably something rounding down the float ranging from 0..1) to zero.

It's essentially doing event.tfinger.x * S.w (and then casting that to a Lua number from its original cdata C type), since Lua numbers are also floats, there shouldn't be any precision loss here (we do get subpixel precision on some Android devices in a veeeery similar codepath, and it works just fine).

I'm... not quite sure where that S.w/S.h comes from, but I'm assuming it's supposed to be the SDL window's dimensions, and I would guess those are 0 here for... some reason?

(EDIT: I'm assuming the actual dx/x & dy/y floats in the actual SDL events are sane?)

[akemnade]

event.tfinger.x/y are zero.

    elseif event.type == SDL.SDL_MOUSEMOTION
        or event.type == SDL.SDL_FINGERMOTION then
        local is_finger = event.type == SDL.SDL_FINGERMOTION
        if is_in_touch then
            if is_finger then
                if event.tfinger.dx ~= 0 then
                    io.write("TF X ", event.tfinger.x, " * ", S.w, "\n")
                    genEmuEvent(C.EV_ABS, C.ABS_MT_POSITION_X,
                        event.tfinger.x * S.w)

Tested with that snippet. S.w/h are sane.

[NiLuJe]

Huh. What about the deltas?

(As I could imagine an optimization where they left computing updated positions to the caller solely based on the delta data?)

[pazos]

debian is like the emulator. It repurposes sdl2 from the target to avoid X11/wayland deps. The rest is the same, with the exception of a couple of fonts.

[akemnade]

I get same values for tfinger.x if I ssh -X desktop on the ebookreader and start koreader on the desktop cpu (=amd64 arch, both sides use debian bullseye).

[NiLuJe]

What about tfinger.dx & tfinger.dy? (If those are sane, dealing with it properly would require having a sane tfinger.x & tfinger.y on FINGERDOWN, though).

[akemnade]

ok, fond the root issue SDL_TouchID and SDL_FingerID size conflict
SDL C api: 64bit
LUA api: 32bit

[akemnade]

I get same values for tfinger.x if I ssh -X desktop on the ebookreader and start koreader on the desktop cpu (=amd64 arch, both sides use debian bullseye).

oops, totally misleading typo
s/same/sane/
because on that 64bit system there is no size mismatch in SDL_TouchID/FingerID

[NiLuJe]

ok, fond the root issue SDL_TouchID and SDL_FingerID size conflict SDL C api: 64bit LUA api: 32bit

Goddamnit. Nice catch.

I went on a hunt for those a while ago, but I obviously missed those, thanks!

[NiLuJe]

NiLuJe/koreader-base@6235941 ought to fix it then ;).

EDIT: Nope, not just yet...

[NiLuJe]

Take two at koreader/koreader-base@9160d36 should actually fix it for realz ^^.

[akemnade]

I build the deb target overnight and installed it. Seems to work now.