Remote client on MacOS is not using ssh-agent and is prompting to unlock keys everytime

[mikewilks]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

The MacOS remote client is not making use of the ssh-agent and is prompting for the password to unlock the SSH key every time podman is used.

Steps to reproduce the issue:

1. brew install podman, config remote podman according to https://www.redhat.com/sysadmin/podman-clients-macos-windows

2. ssh-add key on the Mac

3. run any podman command (podman ps for example) multiple times - each time it will prompt for the unlock password

Describe the results you received:

podman prompts for the unlock password for the SSH key every time

Describe the results you expected:

podman not to prompt for the unlock password and to use the ssh-agent

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 2.1.1

Output of podman info --debug:

Mikes-MacBook:~ mike$ podman info --debug
Key Passphrase: 
host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.21-1.el8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 44dc2e90174f4dcd4040012a62364e7f2564d431-dirty'
  cpus: 12
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: journald
  hostname: removed.host
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.18.0-193.19.1.el8_2.centos.plus.x86_64
  linkmode: dynamic
  memFree: 31318765568
  memTotal: 33448837120
  ociRuntime:
    name: runc
    package: runc-1.0.0-145.rc91.git24a3cf8.el8.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.2-dev'
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-2.el8.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 16877875200
  swapTotal: 16877875200
  uptime: 2h 7m 28.33s (Approximately 0.08 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/mike/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 1
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.el8.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/mike/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 13
  runRoot: /run/user/1000
  volumePath: /home/mike/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1601258926
  BuiltTime: Mon Sep 28 03:08:46 2020
  GitCommit: ""
  GoVersion: go1.13.15
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

client: $ brew info podman
podman: stable 2.1.1 (bottled)

server: $ rpm -q podman
podman-2.1.1-4.el8.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

MacOS Catalina - Version 10.15.6 (19G2021)

[mheon]

@jwhonce PTAL

[rhatdan]

@ashley-cui PTAL

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@ashley-cui Did you get a chance to look at this? Seems pretty key that we add this support.

[ashley-cui]

@rhatdan Haven't taken a look yet, will try to reproduce/do a little digging today

[ashley-cui]

tried looking into this, but ran into this issue: #8323

going to dig further..

[ssbarnea]

I can confirm this bug which renders macos podman client useless, as I do need and want to run containers on my remote linux host, the same way I do with docker.

Let me know if I can help as macos happens to be my main desktop (zbr on irc).

[ssbarnea]

After wasting more than two hours trying to find any way to make podman from macos connect to a linux box I have to surrender. It seams that podman silently fails to load a ssh key even if given unencrypted and debug logging does not help at all:

export CONTAINER_HOST=ssh://root@leno/run/podman/podman.sock
export CONTAINER_SSHKEY=/Users/ssbarnea/.ssh/id_rsa_insecure
podman --log-level=debug info --debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called info.PersistentPreRunE(podman --log-level=debug info --debug)
Login password:
Error: Failed to create sshClient: Connection to bastion host (ssh://root@leno/run/podman/podman.sock) failed.: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain
FAIL: 125

[rhatdan]

@ashley-cui @jwhonce PTAL

[mikewilks]

Just to confirm this is still an issue in Big Sur (11.0.1) with : podman: stable 2.2.0 (bottled) and (server side ) podman-2.1.1-10.el8.x86_64

[ssbarnea]

Yep, it does and combined with #8499 bug creates a bad user experience. Lucky the incorrect "Login" prompt was fixed on Friday but we still have these two issues to address: failure to load RSA keys and failure to reuse agent loaded keys.

@mikewilks Keep in mind that if you see "Login password", this is not the key key password. That bug was fixed in master so you should be able to login without prompt unless you manually specified CONTAINER_SSHKEY.

Apparently as soon you define a particular key using CONTAINER_SSHKEY, you will get a key password prompt from podman regardless the fact that the same key may already be loaded by the ssh agent.

That is annoying as it forces users to make use of unsafe security practices when they have multiple keys. We all know that ssh authentication allows verification of a limited number of ssh keys before the client is rejected, usually 3 or less. Further attempts in short period of time could even ban the client.

This means that you may want to hint the ssh application which particular key to use with each ssh server, to avoid sending wrong keys (also performance impact).

Bad part is that presence of CONTAINER_SSHKEY does disables the ssh agent. CONTAINER_SSHKEY acs mainly like -i option on ssh, which does not disable the use of agent. In fact the only way to disable the agent is to unset SSH_AUTH_SOCK variable.

[jwhonce]

@ssbarnea I don't have a fix for the RSA issue but #8676 prioritizes the ssh-agent keys over a matching --identity provided key. The match is based on the sha256 fingerprints. It concatenates the ssh-agent keys with any key provided by --identity. We've moved to you get ssh-agent support unless you unset SSH_AUTH_SOCK Note --log-level=debug also dumps the ssh keys found and then those rejected as dups.

/cc @ashley-cui is helping me test different remote client setups.

FYI, podman uses the following prompts:

• "Login password" or "'s Login password" when podman is attempting to connect to the remote server and no keys were provided or worked
• "Key Passphrase" when podman is attempting to read a passphrase protected key

[ssbarnea]

Thanks for the updates on these. I already added an ssh-ed25519 to my list of keys so the RSA issue is not really pressing. I would say that the lack of error was likely more annoying than the lack of support for RSA.

I support looking for agent keys before using provided key because this avoids the case where it may ask for key passphrase when in fact the same key was loaded inside the agent. I will try to test the linked change myself and comment on it.

[ashley-cui]

@ssbarnea we have this open right now: #8676

This patch should allow podman to use the ssh-agent. The problem right now is we have a catch 22 (even with the new patch) where we need the passphrase to read the identity file to determine if that file is already in the ssh-agent ... to use without a passphrase.
To get around this (for now) make sure your connection is not associated with an identity file, ie podman system connection add without using the --identity flag and doing a podman system connection ls and making sure there is no identity associated with the connection. Then podman should correctly use the ssh-agent without prompting for the passphrase.

[carrete]

After wasting more than two hours trying to find any way to make podman from macos connect to a linux box I have to surrender

I have a podman server running in an Ubuntu 20.04 multipass vm, and this works on macOS:

podman --remote --url "ssh://ubuntu@${IPV4_ADDR}/run/user/1000/podman/podman.sock"

If I use CONTAINER_HOST and CONTAINER_SSHKEY instead, I'm prompted for a passphrase

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[ssbarnea]

That bot is annoying. the bug was not fixed, and because there was no release made in between I am not able to test the patch.

[rhatdan]

@ssbarnea The bot is necessary to give us a kick in the pants to look at aging issues. Otherwise these issues would just drift away. Everytime one fires I re-read the issue and attempt to get it attention or close the issue if it has been fixed. IE A necessary evil.

[rhatdan]

@baude FYI

[ssbarnea]

@rhatdan As long you do not burndown is may be a good approach. To be honest, I do find podman project as being one of the most dynamic ones.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@baude PTAL

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@baude while in MAC World could you look at this?

[ssbarnea]

I am unable to reproduce this bug with podman 3.0.1 on macos, I have my ssh keys encrypted at rest and if it would not work I would not be able to use podman cli, which I do.

[maciej-trebacz]

@ssbarnea @baude I'm running Podman 3.1.0 and I just stumbled upon it. I have ssh-agent running, I've run ssh-add and entered my passphrase. I can connect to my remote machine without it asking for the passphrase using publickey auth. But doing podman ps (after adding a remote connection) still asks for my passphrase every time.

[tashian]

For posterity, I've run into this in v3.3.1 on macOS. I first ran podman system connection add with the --identity flag. With this option it will ask for a passphrase for that key, even if that key is present in the SSH agent. I needed to re-add the connection without --identity to get podman to use my ssh-agent keys.

[rhatdan]

Could you open a new issue for this, if this is something we should fix.

[ssbarnea]

@tashian I am not sure how you ended up with this because it appears to work for me, and my ssh keys are available only through the agent (they are encrypted at rest).

Maybe the SSH_AUTH_SOCK got lost before podman was called? I recently had a case like this when using tox-ansible and I had to ensure the variable is kept.

[vitaliy-sk]

still reproducible on podman-remote version 3.4.1

podman-remote system connection add xxx --identity ~/.ssh/id_rsa ssh://xxxx/run/user/1000/podman/podman.sock
podman-remote ps
Key Passphrase:

[rhatdan]

Please open an new issue.

[jeremy-chua]

For posterity, I've run into this in v3.3.1 on macOS. I first ran podman system connection add with the --identity flag. With this option it will ask for a passphrase for that key, even if that key is present in the SSH agent. I needed to re-add the connection without --identity to get podman to use my ssh-agent keys.

Yes, omitting "--identity" works