Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

27348363: Mac OS X 10.12: ssh-agent does not automatically load passphrases on the keychain during startup #15361

Open
openradar-mirror opened this issue Jul 14, 2016 · 44 comments

Comments

@openradar-mirror
Copy link

@openradar-mirror openradar-mirror commented Jul 14, 2016

Description

Summary:
In previous versions of macOS, ssh-agent used to remember the passphrases for the keys I added to the keychain with “ssh-add -K ”. After a reboot (or logout/login), it automatically picked up the passphrases from the keychain with no extra step.

In Sierra, I have to manually poke the agent to recognize there are passphrases on the keychain.

Steps to Reproduce:

  1. Create an ssh key with a passphrase
  2. Add its passphrase to the keychain using “ssh-add -K ”. It prompts for the passphrase and adds it to the keychain.
  3. Enjoy passwordless authentication to GitHub, your servers, etc. (“ssh-add -l” confirms ssh-agent knows about the keys.)
  4. Reboot your Mac
  5. “ssh-add -l” shows the agent is empty. Applications using SSH authentication do not work or constantly prompt for a passphrase.
  6. Run “ssh-add -K ” again. It returns immediately without a passphrase prompt, as it finds the correct passphrase already on the keychain.
  7. SSH authentication now works fine.

Expected Results:
The agent should automatically load keys whose passphrases it can access on the keychain.

Actual Results:
I need to manually poke the agent after each logout/login.

Regression:
This used to be automatic before the Sierra betas. On one of my secondary Macs running El Capitan, ssh-agent still remembers SSH keys across reboots, with no special configuration (as far as I can tell).

Notes:

Here is a metadata dump of the two passphrase entries on my keychain. Keychain access shows that their ACLs always allow access to “ssh-agent”, “ssh-add” and “ssh”.

$ security find-generic-password -s "SSH" -a "/Users/lorentey/.ssh/id_rsa"
keychain: "/Users/lorentey/Library/Keychains/login.keychain-db"
version: 512
class: "genp"
attributes:
0x00000007 ="SSH: /Users/lorentey/.ssh/id_rsa"
0x00000008 =
"acct"="/Users/lorentey/.ssh/id_rsa"
"cdat"=0x32303130303932343138333830345A00 "20100924183804Z\000"
"crtr"=
"cusi"=
"desc"=
"gena"=
"icmt"=
"invi"=
"mdat"=0x32303130303932343138333830345A00 "20100924183804Z\000"
"nega"=
"prot"=
"scrp"=
"svce"="SSH"
"type"=

$ security find-generic-password -s "SSH" -a "/Users/lorentey/.ssah/git-fusion"
keychain: "/Users/lorentey/Library/Keychains/login.keychain-db"
version: 512
class: "genp"
attributes:
0x00000007 ="SSH: /Users/lorentey/.ssh/git-fusion"
0x00000008 =
"acct"="/Users/lorentey/.ssh/git-fusion"
"cdat"=0x32303133313131353130353430335A00 "20131115105403Z\000"
"crtr"=
"cusi"=
"desc"=
"gena"=
"icmt"=
"invi"=
"mdat"=0x32303133313131353130353430335A00 "20131115105403Z\000"
"nega"=
"prot"=
"scrp"=
"svce"="SSH"
"type"=

Product Version: Mac OS X 10.12 (16A239j)
Created: 2016-07-14T14:01:35.005190
Originated: 2016-07-14T16:01:00
Open Radar Link: http://www.openradar.me/27348363

@donCarlosOne
Copy link

@donCarlosOne donCarlosOne commented Sep 11, 2016

Confirmed that this problem still exists in OSX 10.12 GM Candidate

@daenney
Copy link

@daenney daenney commented Sep 11, 2016

According to https://twitter.com/lorentey/status/753581927412686850's interaction with Apple this is now the expected behaviour.

Basically the -K is still there and it'll store the thing in the Keychain but it never gets loaded from it.

@zwily
Copy link

@zwily zwily commented Sep 19, 2016

In the past, when you'd create an ssh-agent it would automatically load all keys from the keychain. I use a utility that creates separate agents for each key that I use, so I don't forward personal agents to work servers (for example), and the utility had to start an agent then delete all unexpected keys from it. With this change, I just start an agent and add the desired agent with ssh-add -K /path/to/key and no cleaning up necessary.

Overall, this is probably a good change.

@zenopopovici
Copy link

@zenopopovici zenopopovici commented Sep 19, 2016

Dunno about this... this was also present in last year's beta (10.11) and was fixed in the first bugfix release after launch.

@mgrandi
Copy link

@mgrandi mgrandi commented Sep 20, 2016

running ssh-add -K still asks me for a password when it should be in the keychain. The keychain entry hasn't changed since may of 2015

@jimcistaro
Copy link

@jimcistaro jimcistaro commented Sep 22, 2016

@mgrandi

You should be able to do an ssh-add -A (Add identities to the agent using any passphrases stored in your keychain.). -K is to store it to keychain the first time.

@supryan
Copy link

@supryan supryan commented Sep 23, 2016

This kinda sucks if this was intentional behavior. I liked not having to think about it between reboots. I hope they fix this.

@karlhorky
Copy link

@karlhorky karlhorky commented Sep 24, 2016

Some things that look like they could be helpful:

I'm going to try the ~/.ssh/config file with the AddKeysToAgent option from the first link.

@donCarlosOne
Copy link

@donCarlosOne donCarlosOne commented Sep 26, 2016

I've found that the best way (for me) to deal with this is to execute a script upon opening a terminal window/tab. I use iTerm2, which does this automatically for me.

The script simply runs a basic command:

ssh-add -K .....

followed by each key that I need to have available whilst in a terminal session.

@jbz
Copy link

@jbz jbz commented Oct 5, 2016

I just ran into ssh-agent problems with a new mac running El Cap. I can't use donCarlosOne's solution because my keys have passphrases. Basically, as far as I can tell, launchd is not starting an ssh-agent process at login - so I don't have the env var SSH_AUTH_SOCK set, so nothing can talk to the agent from my login process tree. I have no idea why it's not starting and providing the socket info. This is utterly maddening - I don't care about between reboots (although I'd like the Keychain to mean I don't need to re-add keys to the damn agent) but the agent won't run at all now. It runs if I start it manually with launchctl start, but of course the launchctl getenv SSH_AUTH_SOCK is still null. AIIIGH. Currently troling the net looking for a fix that doesn't involve installing an entire new openssh setup via brew.

@mgrandi
Copy link

@mgrandi mgrandi commented Oct 5, 2016

@jbz when you do ssh-add -K it prompts you for the passphrase, and then it stores it in some magical keychain (that is no longer the Mac OS X keychain), so that when you do ssh-add -A later, it reads said magical keychain and loads them even if they have passwords.

and not sure your problem is related to this thread, ssh_agent should still be using the mac os x keychain (which is what we are talking about), something else is wrong for you =(

@jbz
Copy link

@jbz jbz commented Oct 5, 2016

Yeah, something else is wrong. Sorry, shouldn't have threadjacked. I found a hackaround.

@JJJ
Copy link

@JJJ JJJ commented Dec 15, 2016

Found this issue searching for what I'd hoped would be a solution better than mine. I put this in my ~/.bash_profile:

{ eval `ssh-agent`; ssh-add -A; } &>/dev/null

Now my key gets silently added on every new login shell. If you previously did ssh-add -K and entered your password, you should not be asked for your password again.

@trisweb
Copy link

@trisweb trisweb commented Jan 3, 2017

The above solution by @JJJ, while functional, slows down the loading of each new shell session by 2-3 seconds as the keys are reloaded. Anyone have any ideas for only loading it on the first new login session?

As it stands this will launch and detach the subprocess so your prompt promptly appears:

{ eval `ssh-agent`; ssh-add -A; } &>/dev/null &

However, this means that on your first login session, your keys aren't instantly ready, and also that it's still (somewhat inefficiently) run on each new login session in the background.

Another alternative could be a launchd script that runs on login, which might be pretty bulletproof, but is also a bit complicated to set up. Maybe a quick-bootstrap script would be in order.


Edit: looks like @joshbuchea's answer directly below is the one. Restarted and all keys were loaded appropriately. Great!

@joshbuchea
Copy link

@joshbuchea joshbuchea commented Jan 3, 2017

This solution may be a bit naive, but I was able to restore pre macOS Sierra ssh-agent behavior by creating a config file (~/.ssh/config) with the following contents:

Host *
  UseKeychain yes
  AddKeysToAgent yes
  IdentityFile ~/.ssh/id_rsa

Optionally replace ~/.ssh/id_rsa with the path to your key. To add additional keys, add a new line for each key with: IdentityFile /path/to/your_key.

References:

I summarized this information in a blog post. Critiques and alternative solutions welcome!

@royalaid
Copy link

@royalaid royalaid commented Jan 4, 2017

Can confirm @joshbuchea's solution works, and it appears that you do not need to ssh-add -K either so this really does mimic the old behavior!

@zedd45
Copy link

@zedd45 zedd45 commented Jan 4, 2017

I used @jimcistaro 's solution (ssh-add -A) and it solved my problem on 10.12.2. I already had this stored in the keychain; it was just failing to load it on any new terminal session.

This seems to have only worked until I rebooted, so I assume as soon as the agent shut down this stopped working.

Tried @joshbuchea's solution, and it's working so far (10.12.2). This solution continues to persist (properly) between reboots.

@GWShark0
Copy link

@GWShark0 GWShark0 commented Jan 4, 2017

Also confirming that @joshbuchea's solution works in 10.12.2. Thank you!

@JJJ
Copy link

@JJJ JJJ commented Jan 4, 2017

@joshbuchea's is what I was hoping to find. Thanks!

The key bit is:

UseKeychain yes

Derailing

You're all already using ~/.ssh/config to manage your connections, right? Right?

Now mine starts like:

ControlPersist  1
ControlMaster   auto
ControlPath     ~/.ssh/tmp/%h_%p_%r

Host *
  IdentityFile.  ~/.ssh/id_rsa
  UseKeychain    yes
  AddKeysToAgent yes

Host proxy
  StrictHostKeyChecking no
  LogLevel              QUIET
  HostName              PUBLIC IP OF PROXY SERVER
  HostKeyAlias          PUBLIC IP OF PROXY SERVER
  Port                  22 
  User                  REMOTE USER ID
  IdentityFile          ~/.ssh/id_rsa
  UserKnownHostsFile    ~/.ssh/known_hosts

...which lets all of my other connections look like:

Host chatserver
  HostName PRIVATE IP OF CHAT SERVER

...and ensures all subsequent connections go through the proxy server first, and none of the servers behind the proxy need to be publicly accessible through SSH.

Then I can ssh proxy to connect to the proxy server directly, or ssh chatserver to connect to the chat server via the proxy server.

Single point of failure? Sure, but I think it's better than having a bunch of publicly open SSH ports, or remembering what port numbers you tried to obfuscate them to.

/derail, and thanks again for this @joshbuchea

@matthewberkompas
Copy link

@matthewberkompas matthewberkompas commented Jan 5, 2017

@joshbuchea's solution worked great for me as well. Thanks for posting!

@KrisJanssen
Copy link

@KrisJanssen KrisJanssen commented Jan 12, 2017

Out of interest,

Many users here have indeed reported that ssh-add -K indeed no longer adds the passphrase for your keys into the regular keychain. I can confirm that you cannot find any entries in Keychain app, nor through the command line:

OSX01:~ kris$ security find-generic-password -s "SSH" -a "/Users/kris/.ssh/id_rsa" keychain: "/Users/kris/Library/Keychains/login.keychain-db" security: SecKeychainSearchCopyNext: The specified item could not be found in the keychain.

The question I than have is, where is it going instead because the passphrase is still somehow magically stored given that some of the above solutions work... also for me, creating an ssh config file restores the expected behaviour so ssh-add -K is still doing something.

@alexventuraio
Copy link

@alexventuraio alexventuraio commented Jan 15, 2017

I confirm @joshbuchea's solution works! I'm running Mac OS X Sierra 10.12.2 (16C68)

@andreorvalho
Copy link

@andreorvalho andreorvalho commented Jan 22, 2017

I can also confirm @joshbuchea's solution works in Mac OS X Sierra 10.12.2

@mgrandi
Copy link

@mgrandi mgrandi commented Jan 23, 2017

@KrisJanssen it seems to be stored in a sqlite3 database at ~/Library/Keychains/<SOME-UUID>/keychain-2.db , in the genp table, you can search for com.apple.ssh.passphrases in the agrp column

@KrisJanssen
Copy link

@KrisJanssen KrisJanssen commented Jan 24, 2017

@mgrandi very cool, thanks for the tip!

dchabot pushed a commit to dchabot/dotfiles that referenced this issue Jan 27, 2017
Alhadis added a commit to Alhadis/.files that referenced this issue Feb 1, 2017
@somecallmejosh
Copy link

@somecallmejosh somecallmejosh commented Feb 2, 2017

👍 - @joshbuchea's solution worked for me, too... OSX Sierra 10.12.3

@woniesong92
Copy link

@woniesong92 woniesong92 commented Apr 3, 2017

When I tried the following

Host *
  UseKeychain yes
  AddKeysToAgent yes
  IdentityFile ~/.ssh/id_rsa

I get

/etc/ssh/ssh_config: line 104: Bad configuration option: usekeychain
/etc/ssh/ssh_config: line 105: Bad configuration option: addkeystoagent

Anyone else having the issue on El Capitan, 10.11.16?

@suresh7745
Copy link

@suresh7745 suresh7745 commented Apr 5, 2017

i see the same error as above

@les-okta
Copy link

@les-okta les-okta commented Apr 5, 2017

@woniesong92 and @suresh7745 those two options are not available on El Capitan. Run man ssh_config to see the available options. UseKeychain and AddKeysToAgent are new to OS 10.12 Sierra.

@o6uoq
Copy link

@o6uoq o6uoq commented Apr 6, 2017

I added UseKeychain yes to my ~/.ssh/config but for some reason, it made ssh-agent keep falling over.

I commented it out and now just run ssh-add -A after every Mac reboot.

Anyone else experienced a similar issue with ssh-agent while using UseKeychain ?

@les-okta
Copy link

@les-okta les-okta commented Apr 6, 2017

@o6uoq you have to also add AddKeysToAgent yes. For example:

Host *
  UseKeychain yes
  AddKeysToAgent yes
@o6uoq
Copy link

@o6uoq o6uoq commented Apr 6, 2017

@les-okta I had that set, as below:

AddKeysToAgent yes
UseKeychain yes

..but I commented out UseKeychain as for some reason it made ssh-agent fall over. I've not spent much time troubleshooting, but commenting it out and using ssh-add -A has been my temp work around.

@les-okta
Copy link

@les-okta les-okta commented Apr 6, 2017

They have to be on separate lines, like my example above.

@o6uoq
Copy link

@o6uoq o6uoq commented Apr 6, 2017

@les-okta they are, that was just a typo above - same problem.

@nalexn
Copy link

@nalexn nalexn commented Apr 27, 2017

@les-okta
Even though I'm on OS X 10.12.4 I still get

Bad configuration option: usekeychain
Bad configuration option: addkeystoagent

I've noticed I have two paths where ssh, ssh-add and ssh-agent are stored:

/usr/local/bin/
/usr/bin/

and apparently they are of different versions, because

ssh-add -K id_rsa

tells me there is no option -- K, meanwhile

/usr/bin/ssh-add -K id_rsa

works with no issues.

Update

It turned out in /usr/local/bin/ there are symbolic links for all the ssh executables. I updated the sym links to point /usr/bin/..., and it solved the problem for me:

ln -sfn /usr/bin/ssh /usr/local/bin/ssh
ln -sfn /usr/bin/ssh-add /usr/local/bin/ssh-add
ln -sfn /usr/bin/ssh-agent /usr/local/bin/ssh-agent
ln -sfn /usr/bin/ssh-keygen /usr/local/bin/ssh-keygen
ln -sfn /usr/bin/ssh-keyscan /usr/local/bin/ssh-keyscan

@orderthruchaos
Copy link

@orderthruchaos orderthruchaos commented May 1, 2017

After doing ssh-add -K manually for each key (using absolute paths), you could add this to your .bashrc:

if [[ $- = *i* ]]; then
  declare -i __agent_run_state=$(ssh-add -l >| /dev/null 2>&1; echo $?)
  if [ "$SSH_AUTH_SOCK" -a $__agent_run_state -eq 1 ]; then
    ssh-add -A
  fi
  unset __agent_run_state
fi

You will want to make sure that the key paths are absolute in your keychain.

lars-derichter added a commit to lars-derichter/mydotfiles that referenced this issue May 5, 2017
…rt/openradar-mirror#15361 joshbuchea's answer
@mykmelez
Copy link

@mykmelez mykmelez commented Aug 28, 2017

It turned out in /usr/local/bin/ there are symbolic links for all the ssh executables. I updated the sym links to point /usr/bin/..., and it solved the problem for me:

@nalexn You probably have another installation of SSH (perhaps via MacPorts or Homebrew). Instead of updating the symlinks, uninstall the other installation (which is wherever those symlinks pointed to). That should remove the symlinks themselves as well. If it doesn't, you can simply remove them yourself, and then the SSH executables in /usr/bin will become the first ones on your PATH, and you'll use them by default.

@RyanHendry-awin
Copy link

@RyanHendry-awin RyanHendry-awin commented Oct 16, 2017

I had to brew uninstall openssh to get the above ssh config to work.

@tea-spoon
Copy link

@tea-spoon tea-spoon commented Feb 19, 2018

Thanks for this @joshbuchea @chaintip!

@chaintip
Copy link

@chaintip chaintip commented Feb 19, 2018

@joshbuchea has claimed the 0.00327139 BCH| ~ 3.74 USD sent by @tea-spoon via chaintip.

@joshbuchea
Copy link

@joshbuchea joshbuchea commented Feb 24, 2018

Thanks for the BCH @tea-spoon! 🙏

@jeffreyplusplus
Copy link

@jeffreyplusplus jeffreyplusplus commented Jun 23, 2018

@woniesong92
Adding IgnoreUnknown UseKeychain before UseKeychain yes solved the problem for me.

@o6uoq
Copy link

@o6uoq o6uoq commented Jun 24, 2018

Does anyone have this successfully working with brew ? I've tried all the things and nothing works other than ssh-add -A after a reboot. Could my Keychain be possibly corrupt? Or, is it an known issue that brew + OpenSSH = no KeyChain working?

@joshbuchea
Copy link

@joshbuchea joshbuchea commented Jun 25, 2018

@o6uoq looks like above ^^ @RyanHendry-awin needed to uninstall OpenSSH via Brew to resolve:

had to brew uninstall openssh to get the above ssh config to work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.