[BUG] Selkies-based UI missing core functionality for usability from iOS based devices which KasmVNC provides

[Chirishman]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I see the potential benefits of the Selkies based version and it does seem to be mostly a straight upgrade on desktop web browsers but several specific behaviors make this completely unusable for my workflow/usage pattern:

I will break them down into two basic categories although the issues with each compound the other:

Resolution Problems

• Automatic resolution is setting an improper height on iOS cutting the title bar and the status bar at the bottom of the Calibre UI off. This includes various "yes" and "Continue" dialogue buttons which always appear in the bottom right corner. This also removes access to the horizontal scroll bar.
• Resolution detection by Selkies is non-deterministic. Rotating the display from landscape to portrait and back while set to fullscreen results in a different resolution from the originally displayed resolution
• Clicking "reset to window" button after trying a preset resolution sometimes results in an all-black screen that can only be recovered by refreshing the page or (for some reason) toggling the HiDPI (Pixel Perfect) toggle

User Input Deficiencies

• Selkies, unlike KasmVNC offers no touch-based way to input modifier key inputs such as CTRL, ALT, ESC, TAB, etc, disallowing the triggering of said input dialogues with CTRL+Y, or Tabbing to the correct input and hitting spacebar, or similar. This also breaks my workflow for this tool which involves CTRL+Click selecting non-contiguous books to mark them as having been read/completed
• Toggling the Onscreen Keyboard exits fullscreen mode every time
• The mouse cursor/trackpad behavior option is not properly compatible with fullscreening on iOS. Because the system gesture for exiting fullscreened browser playback is pulling down if any of your mouse gestures start with a downward movement it exits fullscreen mode immediately.
• Stylus mouse input (which works fine on KasmVNC) is fully unusable in Selkies unless you disable writing-based text input system-wide on your iOS Device.

Impact

My daily use of this container includes

• Open at the default normal resolution
• ctrl+click selecting several items by tapping them with my stylus
• using a ctrl shortcut to mark them as read
• going to a different tab in my Calibre library, scrolling sideways to check a few columns
• hitting ctrl+a to select all and clicking a button to cause a plugin to check for new chapters/updates on the web serials I'm following
• Clicking on the "Jobs" tracker in the bottom right corner to open the window to check on the long running job's progress
• Inputting ESC to close the window using the KasmVNC touch UI
• Clicking Yes on the UI dialogue in the bottom right to accept the changes to my library/updates to my downloaded books

I have bolded the items of my daily use of this container which are difficult or not possible with the current version of Selkies.

Here is a screen recording demonstrating the cut off UI elements, mouse input issues, etc on Chrome for iOS on an M4 iPad Pro

Expected Behavior

• Proper resolution/viewable area detection on iOS Chrome, resulting in UI elements such as horizontal scrollbars, titlebars, and confirmation dialogue buttons not being cut off
• Deterministic resolution detection not resulting in a different resolution before and after screen rotation
• Ability to input CTRL, ALT, TAB, ESC

Steps To Reproduce

1. Install the current version
2. Attempt to access UI elements at the bottom of the window on iOS Chrome
3. Attempt to input CTRL, ALT, TAB, ESC

Environment

- OS: Ubuntu 24.04.2 LTS
- How docker service was installed:
Not relevant

CPU architecture

x86-64

Docker creation

version: '3'

volumes:
  calibre_config:
    driver: local
    driver_opts:
      type: none
      device: "/var/containers/dockerconfigs/calibre"
      o: bind
  litshare:
    driver: local
    driver_opts:
      type: cifs
      device: //10.1.2.20/LitShare/
      o: addr=10.1.2.20,username=REDACTED,password=REDACTED,uid=1000,gid=1000,vers=3.0,file_mode=077>
services:
  calibre:
    image: lscr.io/linuxserver/calibre:latest
    deploy:
      placement:
        constraints:
          - "node.labels.role==nas"
#    security_opt:
#      - seccomp:unconfined #optional
    environment:
      - PUID=1001
      - PGID=1002
      - TZ=America/New_York
#      - PASSWORD= #optional
#      - CLI_ARGS= #optional
    volumes:
      - calibre_config:/config
      - /mnt/nvme1/literary:/literary
      - litshare:/litshare
    ports:
      - 8180:8080
      - 8181:8181
      - 8081:8081

Container logs

Not relevant

[github-actions]

Thanks for opening your first issue here! Be sure to follow the relevant issue templates, or risk having this issue marked as invalid.

[github-actions]

Thanks for opening your first issue here! Be sure to follow the relevant issue templates, or risk having this issue marked as invalid.

[thelamer]

Modifier keys on mobile is known and being worked on.
#44

As for resolution cutoff and safe area on mobile I will work on that.

[thelamer]

I feel gross but I ordered an iPad, I will work on what I can on mobile, but this will likely take a week to get all sorted out.

[thelamer]

Soft keys are in still working through other issues
selkies-project/selkies@455542f

[thelamer]

I need your help to test this until I get an iPad(its shipping), every container is a development container and you can run head code by doing this:

git clone https://github.com/selkies-project/selkies.git
cd selkies
docker run --rm -it \
  -p 8181:8181 \
  -e PUID=1000 \
  -e PGID=1000 \
  --shm-size=1gb \
  -e DEV_MODE=selkies-dashboard \
  -v $(pwd):/config/src \
  linuxserver/calibre bash

If you cannot run this command and connect I understand it is just that with your results it will speed up the process.

• Automatic resolution is setting an improper height on iOS cutting the title bar and the status bar at the bottom of the Calibre UI off. This includes various "yes" and "Continue" dialogue buttons which always appear in the bottom right corner. This also removes access to the horizontal scroll bar.

this should be fixed and the problem was the viewport calc was using 100vh this does not work on mobile I could replicate this on android

• Resolution detection by Selkies is non-deterministic. Rotating the display from landscape to portrait and back while set to fullscreen results in a different resolution from the originally displayed resolution

this is likely not fixed, it looks like iOS handles screen rotation differently I will need an iPad to do that proper otherwise I am just guessing

• Clicking "reset to window" button after trying a preset resolution sometimes results in an all-black screen that can only be recovered by refreshing the page or (for some reason) toggling the HiDPI (Pixel Perfect) toggle

this should be fixed I tore apart the async code to stop any and all race conditions related to rendering my testing method is reloading the window at a 100ms tic 1000 times.

• Selkies, unlike KasmVNC offers no touch-based way to input modifier key inputs such as CTRL, ALT, ESC, TAB, etc, disallowing the triggering of said input dialogues with CTRL+Y, or Tabbing to the correct input and hitting spacebar, or similar. This also breaks my workflow for this tool which involves CTRL+Click selecting non-contiguous books to mark them as having been read/completed

the soft buttons are now in the sidebar. please test and confirm

• Toggling the Onscreen Keyboard exits fullscreen mode every time

this one is puzzling, but will need an ipad to test. This is likely not fixed.

• The mouse cursor/trackpad behavior option is not properly compatible with fullscreening on iOS. Because the system gesture for exiting fullscreened browser playback is pulling down if any of your mouse gestures start with a downward movement it exits fullscreen mode immediately.

need an ipad for this we do multiple preventdefault actions on touch input but I would need raw input output to actually see it and prevent it(again need iPad)

• Stylus mouse input (which works fine on KasmVNC) is fully unusable in Selkies unless you disable writing-based text input system-wide on your iOS Device.

Will I be able to test this with a base 11 inch iPad? Or do you need an apple pencil or something ?

[Chirishman]

Thanks! I will test tomorrow on my lunch break, I've currently pinned my homelab calibre container to the last Kasm version at the moment.

As far as the stylus thing an Apple Pencil is required to provoke the behavior but all iPads, including the base model from the 6th gen and up should support at least one of the four versions of the Apple Pencil. Third party styluses that can emulate an Apple pencil do also invoke this behavior.

Like I said it appears to be that something is causing the entire view area to be treated as a scribble zone/text box and so any motion of the stylus other than a tap is processed as a handwriting stroke and the OS is attempting to recognize it as writing and turn it into character input. I'll get a screen recording of what it looks like when I roll my container forward tomorrow.

Here's the Pencil-to-iPad compatibility list: https://support.apple.com/en-us/108937

[thelamer]

Ahh that makes sense, we don't capture input from a canvas like Kasm we use an input div overlay, this has advantages and disadvantages, but that particular one might not be fixable without like a hard flag changing modes.
Need to test if we can detect a stylus, they have a "touchType" var unique to their web env so it might be possible in combination with preventdefault wont know until I have hardware.

[Chirishman]

Not sure what I've done wrong here but it's throwing this error when I run the container as instructed:

Blocked request. This host ("calibre.chiriserv.com") is not allowed.
To allow this host, add "calibre.chiriserv.com" to `server.allowedHosts` in vite.config.js.

I can't find the referenced file anywhere in the source of this repo or in any of the folders mounted to the container, please advise on how to proceed.

Here's the modded compose file I'm using:

volumes:
  calibre_config:
    driver: local
    driver_opts:
      type: none
      device: "/var/containers/dockerconfigs/calibre"
      o: bind
  litshare:
    driver: local
    driver_opts:
      type: cifs
      device: //10.1.2.20/LitShare/
      o: addr=10.1.2.20,username=REDACTED,password=REDACTED,uid=1000,gid=1000,vers=3.0,file_mode=0777,dir_mode=0777

services:
  calibre:
    #image: linuxserver/calibre:v8.7.0-ls344
    image: linuxserver/calibre:latest
    shm_size: 1gb
    deploy:
      placement:
        constraints:
          - "node.labels.role==nas"
    environment:
      - PUID=1001
      - PGID=1002
      - TZ=America/New_York
      - DEV_MODE=selkies-dashboard
    volumes:
      - calibre_config:/config
      - /mnt/nvme1/literary:/literary
      - litshare:/litshare
      - /var/containers/dockerconfigs/selkies:/config/src
    ports:
      - 8180:8080
      - 8181:8181
      - 8081:8081

[thelamer]

just pull the latest container it has the current state of the app.

[Chirishman]

OK so the results of my testing are:

• Viewable area problems seem fixed!
• The hitting "Ctrl" or other modifiers in the sidebar overlay dismisses the virtual keyboard which is very annoying
• The Ctrl+A key combo to select all doesn't work, it pops the add-book dialogue which is what happens when you hit the A key with no modifiers.
• The other buttons also seem to work as individual button presses but not keycombo with onscreen keyboard input (shift+tab doesn't go backwards and also dismisses the keyboard again)
• Ctrl+Click input does appear to be working properly

Video of stylus behavior and other described behavior here

[thelamer]

Thanks for the feedback!
That is unfortunate about all the keyboard popping stuff hopefully that is an easy fix.
I was hoping there would be at least one thing I could look at in an android env but these all look like iOS specific, also I live in the middle of nowhere so looks like Saturday is the actual start time on having the ability to close these out.