remote_file does not work when providing ssh key through flags

[Jackenmen]

Describe the bug
A clear and concise description of what the bug is.

To Reproduce

1. SSH into the VPS:

ubuntu@local: ~$ kitty +kitten ssh USERNAME@IP_ADDRESS -i .ssh/MY_KEY.pem

2. Run ls:

USERNAME@vps: ~$ ls --hyperlink=auto
some_file.txt

3. Click on a file in the listing while holding Ctrl+Shift.
4. Hit e key
5. See output:

Traceback (most recent call last):
  File "kittens/remote_file/main.py", line 211, in main
  File "kittens/remote_file/main.py", line 289, in handle_action
  File "kittens/remote_file/main.py", line 125, in __enter__
  File "subprocess.py", line 373, in check_call
subprocess.CalledProcessError: Command '['ssh', '-o', 'ControlPath=~/.ssh/kitty-master-8792-%r@%h:%p', '-o', 'TCPKeepAlive=yes', '-o', 'ControlPersist=yes', '-o', 'ControlMaster=auto', '-fN', 'USERNAME@IP_ADDRESS']' returned non-zero exit status 255.
Failed with unhandled exception

Press any key to exit...

Screenshots
N/A

Environment details

kitty 0.23.1 (cd7b4fcd8e) created by Kovid Goyal
Linux ubuntu-virtual-machine 5.11.0-36-generic #40~20.04.1-Ubuntu SMP Sat Sep 18 02:14:19 UTC 2021 x86_64
Ubuntu 20.04.3 LTS \n \l
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"
Running under:X11
Loaded config files:
  /home/ubuntu/.config/kitty/kitty.conf

Config options different from defaults:
Changed shortcuts:
	shift+ctrl+f KeyAction('launch', ('--type=overlay', '--stdin-source=@screen_scrollback', '/usr/bin/fzf', '--no-sort', '--no-mouse', '--exact', '-i', '--tac'))

Additional context

Try to reproduce the problem with kitty --config NONE

This can be reproduced under kitty --config NONE too.

[Jackenmen]

On second look, I see this note in the documentation:

If you have not setup automatic password-less SSH access, then, when editing starts you will be asked to enter your password just once, thereafter the SSH connection will be re-used.

I assume this is the reason why it doesn't work, is there some workaround for this or do I just need to settle for adding the configuration to .ssh/config?