Speak selection on macOs always reads the first word of the last line

[blubber]

Describe the bug
macOs features an accessibility option known as 'Speak Selection', it works by selecting a portion of text followed by pressing a hotkey (default is Option-Esc), the macOs tts engine is used to read the selected text aloud. In Kitty this doesn't work properly, it always reads the first word of the current line. For example, my prompt featuers the current working directory as the first item, which results in it always reading the cwd aloud.

It seems that the selection is not properly conveyed to the OS.

To Reproduce
Steps to reproduce the behavior:

1. Go to System Preferences > Accessibility > Spoken content
2. Enable 'Speak selection'.
3. Select some text (other than the first word in your prompt) in Kitty and press the Speak selection hotkey (defaults to Option-Esc).
4. It will speak the first word of the prompt on the current line, instead of the selected text.

Environment details

kitty 0.25.2 created by Kovid Goyal
Darwin macbookpro.local 21.6.0 Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:22 PDT 2022; root:xnu-8020.140.41~1/RELEASE_ARM64_T6000 arm64
ProductName:	macOS ProductVersion:	12.5 BuildVersion:	21G72
Frozen: True
Paths:
  kitty: /Applications/kitty.app/Contents/MacOS/kitty
  base dir: /Applications/kitty.app/Contents/Resources/kitty
  extensions dir: /Applications/kitty.app/Contents/Resources/Python/lib/kitty-extensions
  system shell: /bin/zsh
Loaded config files:
  /Users/***/.config/kitty/kitty.conf

Config options different from defaults:

Important environment variables seen by the kitty process:
	PATH                                /Applications/kitty.app/Contents/MacOS:/usr/bin:/bin:/usr/sbin:/sbin
	LANG                                en_US.UTF-8
	SHELL                               /bin/zsh
	USER                                ***

Additional context
I also tried this with other shells than ZSH, it results in the same issue. For instance in bash, my prompt starts with bash-3.2. The speak selection voice will read 'bash` regardless of the actual selection.

[kovidgoyal]

I'm afraid I dont care about this, it will require some special purpose
cocoa api or the other and given I dont use apple products the effort
involved is too large for the end result. But patches are welcome. If
you or anyone else wants to implement this feel free to comment and I
will provide any needed guidance.

[page-down]

Content in kitty is rendered by GLFW (OpenGL), not by native Cocoa or other UI frameworks integrated with system accessibility.
So the system doesn't know exactly what text was selected in kitty. Implementing this requires more work.

https://developer.apple.com/documentation/appkit/nsaccessibility?language=objc

I don't think this is a bug, but a feature request.

[kovidgoyal]

yes you will need to find whatever api informs the OS of the current
primary selection, then implement _glfwPlatformSetPrimarySelectionString
in the cocoa backend to call that API. It should be a ten line patch,
the hard part is figuring out what coca api to use, and testing it.

[blubber]

I'll take a look at it myself, at least the above hints should point me in the right direction, because I had no idea whatsoever where to start.

[page-down]

@blubber

Instead of looking up the API in Apple's awful docs, I believe it would be easier to use the following configuration.

kitty -o 'map f1 launch --type=background say -- @selection'

Select a word and press F1.

[blubber]

The problem with that is that say doesn't have access to the siri voices, and listening to the old-school tts voice all day is tiring.

[page-down]

@kovidgoyal

It looks like the accessibility framework will read the selected text instead of being actively pushed by kitty. (Only for the feature mentioned by OP)

Is it possible to get the selected text of the (last) active kitty window in current OS window in glfw/cocoa_window.m?
Executing Speak selection after switching windows will get the text correctly.

[kovidgoyal]

On Sat, Aug 06, 2022 at 02:59:41AM -0700, page-down wrote: @kovidgoyal It looks like the accessibility framework will read the selected text instead of being actively pushed by kitty. (Only for the feature mentioned by OP)

How does the accessibility framework know what text is selected? kitty or rather glfw has to tell it no?

Is it possible to get the selected text of the (last) active kitty window in current OS window in `glfw/cocoa_window.m`? Executing `Speak selection` after switching windows will get the text correctly.

just implement glfwplatformsetprimaryselection, that should be good enough to cover most use cases.

[page-down]

How does the accessibility framework know what text is selected? ...

According to my experiments, the accessibility framework will call the following method (when enabled).

- (NSString *)accessibilitySelectedText {
    return @"the selected text in active kitty window";
}

... _glfwPlatformSetPrimarySelectionString ...

Well, even implementing a fake primary selection (store and read when needed) doesn't look any better than using launch say -- @selection, for users who don't need Siri voice. :)

[kovidgoyal]

On Sat, Aug 06, 2022 at 03:48:52AM -0700, page-down wrote: > How does the accessibility framework know what text is selected? ... According to my experiments, the accessibility framework will call the following method (when enabled). ```objc - (NSString *)accessibilitySelectedText { return @"the selected text in active kitty window";

You can easily add a callback to get the current selected text, it will require a fair bit of boiler plate to pass it through glfw.

[kovidgoyal]

I added API to glfw to fetch the current active selection, just FYI.
4dd696e

[blubber]

I finally had some spare time to take a closer look at this when I saw @kovidgoyal 's commits and @page-down 's PR. I can confirm that it works and I want to thank you folks for taking a look at this even though it got shot down initially. This makes kitty usable for me as a visually impaired user. Great work! 👍