focus_visible_window breaks if interrupted by mouse click

[ecerulm]

Describe the bug
kitty 0.34.1 on macOS Sonoma 14.4.1

When using focus_visible_window , after the numbers are shown over the windows, if I use the mouse to click on a window. kitty will "hang", I won't be able to type in any of the windows.

To Reproduce
Steps to reproduce the behavior:

1. start kitty fresh /
2. kitten @ goto-layout grid
3. kitten @ new-window
4. kitten @ new-window # we need at least 3 windows
5. Run cat on one window just to have some program running
6. F7 (I have `map f7 focus_visible_window)
7. The window number overlay will be shown
8. Use the mouse to click on another window
9. now type, I will just hear a "beep" on every keypress, but the keypress does not seem to reach the shell
10. I can mouse click on other windows and the cursor blinks in the window selected but still keypresses are not sent
11. Close kitty, seems to close ok. (I even get the "Are you sure you want to close this OS window? It is running the xxx program", and I'm able to click on Yes, but not press Y)

Screenshots
If applicable, add screenshots to help explain your problem.

Environment details

kitty 0.34.1 (7f61f1f9f3) created by Kovid Goyal
Darwin Ruben-Laguna 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:11:05 PDT 2024; root:xnu-10063.101.17~1/RELEASE_X86_64 x86_64
ProductName:		macOS ProductVersion:		14.4.1 BuildVersion:		23E224
Frozen: True
Paths:
  kitty: /Applications/kitty.app/Contents/MacOS/kitty
  base dir: /Applications/kitty.app/Contents/Resources/kitty
  extensions dir: /Applications/kitty.app/Contents/Resources/Python/lib/kitty-extensions
  system shell: /bin/zsh
Loaded config files:
  /Users/rubelagu/.config/kitty/kitty.conf

Config options different from defaults:
allow_remote_control               yes
bold_font                          JetBrains Mono ExtraBold
bold_italic_font                   JetBrains Mono ExtraBold Italic
copy_on_select                     clipboard
disable_ligatures                  2
enabled_layouts                    ['tall:bias=50;full_size=1;mirrored=false', 'horizontal', 'stack', 'grid']
font_family                        JetBrains Mono
font_features:
{'FiraCode-Bold': ('+cv02', '+ss03', '+ss07'),
 'FiraCode-Light': ('+cv02', '+ss03', '+ss07'),
 'FiraCode-Medium': ('+cv02', '+ss03', '+ss07'),
 'FiraCode-Regular': ('+cv02', '+ss03', '+ss07'),
 'FiraCode-Retina': ('+cv02', '+ss03', '+ss07'),
 'FiraCode-SemiBold': ('+cv02', '+ss03', '+ss07'),
 'JetBrainsMono-BoldItalic': ('+zero',),
 'JetBrainsMono-ExtraBold': ('+zero',),
 'JetBrainsMono-ExtraBoldItalic': ('+zero',),
 'JetBrainsMono-Italic': ('+zero',),
 'JetBrainsMono-Regular': ('+zero',)}
font_size                          13.0
italic_font                        JetBrains Mono Italic
listen_on                          unix:/tmp/mykitty
macos_option_as_alt                2
macos_quit_when_last_window_closed True
shell                              /bin/zsh --login --interactive
symbol_map:
	U+23fb - U+23fe → Symbols Nerd Font Mono
	U+2665 - U+2665 → Symbols Nerd Font Mono
	U+26a1 - U+26a1 → Symbols Nerd Font Mono
	U+2b58 - U+2b58 → Symbols Nerd Font Mono
	U+e000 - U+e00a → Symbols Nerd Font Mono
	U+e0a0 - U+e0a3 → Symbols Nerd Font Mono
	U+e0b0 - U+e0d4 → Symbols Nerd Font Mono
	U+e200 - U+e2a9 → Symbols Nerd Font Mono
	U+e300 - U+e3e3 → Symbols Nerd Font Mono
	U+e5fa - U+e6aa → Symbols Nerd Font Mono
	U+e700 - U+e7c5 → Symbols Nerd Font Mono
	U+ea60 - U+ebeb → Symbols Nerd Font Mono
	U+f000 - U+f2e0 → Symbols Nerd Font Mono
	U+f300 - U+f32f → Symbols Nerd Font Mono
	U+f400 - U+f4a9 → Symbols Nerd Font Mono
	U+f500 - U+f8ff → Symbols Nerd Font Mono
	U+f0001 - U+f1af0 → Symbols Nerd Font Mono
visual_bell_duration               0.2
Added shortcuts:
	ctrl+. →  layout_action bias 50 62 70
	ctrl+/ →  layout_action mirror toggle
	f1 →  toggle_layout stack
	f7 →  focus_visible_window
Removed shortcuts:
	cmd+, →  edit_config_file
	cmd+enter →  new_window
	cmd+h →  hide_macos_app
	cmd+k →  clear_terminal to_cursor active
	cmd+m →  minimize_macos_window
	cmd+n →  new_os_window
	cmd+q →  quit
	cmd+t →  new_tab
	cmd+w →  close_tab
	ctrl+cmd+, →  load_config_file
	ctrl+cmd+f →  toggle_fullscreen
	opt+cmd+h →  hide_macos_other_apps
	opt+cmd+r →  clear_terminal reset active
	opt+cmd+s →  toggle_macos_secure_keyboard_entry
	shift+cmd+/ →  open_url https://sw.kovidgoyal.net/kitty/
	shift+cmd+[ →  previous_tab
	shift+cmd+] →  next_tab
	shift+cmd+d →  close_window
	shift+cmd+w →  close_os_window

Important environment variables seen by the kitty process:
	PATH                                /Applications/kitty.app/Contents/MacOS:/usr/bin:/bin:/usr/sbin:/sbin
	LANG                                en_US.UTF-8
	SHELL                               /bin/sh
	DISPLAY                             /private/tmp/com.apple.launchd.hrQvMCN20U/org.xquartz:0
	USER                                ruben.laguna@xxxx.com

Additional context

I did reproduced using kitty --config NONE ( using the default control+shift+f7 to switch to the grid layout and performing the same actions described above