plist not loaded on macOS 10.14 Mojave

[lukaskuzmiak]

It seems macOS 10.14 broke the custom plist loading. I compared 10.13.6 on a different device with fresh install of 10.14 and found that while launchctl prioritizes ~/Library/LaunchAgents/ssh-askpass.plist on 10.13.6, on 10.14 it loads the /System/Library/LaunchAgents/com.openssh.ssh-agent.plist plist first (this happens on-demand with first process trying to talk to the agent rather than on-login, which seems to be a consistent behavior to 10.13.6). This then looks like:

$ launchctl list | grep ssh
1036	0	com.openssh.ssh-agent
-	0	com.github.theseal.ssh-askpass
$

And obviously ssh-askpass is never invoked cause of the missing environment variable(s).

The only (ugly) workaround I found is to reboot into recovery, csrutil disable, edit /System/Library/LaunchAgents/com.openssh.ssh-agent.plist and add the environment variables there, back to recovery, csrutil enable.

I tried playing around with launchctl disable or launchctl remove to get rid of the system/com.openssh.ssh-agent but unsuccessfully.

Does anybody have some insight into why this worked on 10.13.6 and if this is really a change of behavior in 10.14 and how launchctl loads stuff?

[theseal]

I used to have a similar problem, but not sure in which version of macOS that was. I couldn't reproduce it on other macs and could work around it by disabling other auto-starting application and booting without a network connection. Thought it was a race condition of some kind. Solved by time (or a minor macOS upgrade…).

Seems like you are using the manual installation. I upgraded my computer to Mojave yesterday and ssh-askpass loaded from ~/Library/LaunchAgents/homebrew.mxcl.ssh-askpass.plist works just fine for me. That file is handled by brew services as stated in the caveats. Can you try the homebrew installation method and see if it works any better?

[lukaskuzmiak]

yes, I've used the manual method. I'm now upgrading my other machine to 10.14 so I'll try it there as the update finishes and let you know (I can try the homebrew method there).

I'll try to debug it further, I'm missing one thing about this - do you have a clue how the "on-demand" execution of ssh-agent works? Like how is it triggered exactly. The plist is loaded on login I suppose but ssh-agent only appears in process list once something (ssh-add or ssh or other process) somehow tries to talk to it. I'm hoping that might help come up with a solution that would avoid the race condition.

Btw, it also happened once on the 10.13.6 during my testing that the system plist was loaded first, i logged out, logged back in and it loaded the user one after .. so it seems like it's pretty .. fragile :)

[theseal]

I'll try to debug it further, I'm missing one thing about this - do you have a clue how the "on-demand" execution of ssh-agent works? Like how is it triggered exactly. The plist is loaded on login I suppose but ssh-agent only appears in process list once something (ssh-add or ssh or other process) somehow tries to talk to it. I'm hoping that might help come up with a solution that would avoid the race condition.

Sorry but I have no idea how the magic works. 🙁 If you find something useful I'm all ears.

This issue also appeared in my 10.14.0 when I logged out and then logged in again. After a reboot everything was fine again. Can't reproduce it again… Hello race condition.

[simmel]

On Tue, 2018-09-25 at 00:55:07 -0700, Lukas Kuzmiak wrote: […] I'll try to debug it further, I'm missing one thing about this - do you have a clue how the "on-demand" execution of `ssh-agent` works? Like how is it triggered exactly. The plist is loaded on login I suppose but `ssh-agent` only appears in process list once something (`ssh-add` or `ssh` or other process) somehow tries to talk to it.

If I've understood it correctly; launchd sets the environment variable SSH_AUTH_SOCK to a "listen" socket which launchd listens on until someone tries to talk to the socket. launchd then starts the application specified in the plist and redirect e.g. ssh-add's communication to ssh-agent.

I'm hoping that might help come up with a solution that would avoid the race condition. Btw, it also happened once on the 10.13.6 during my testing that the system plist was loaded first, i logged out, logged back in and it loaded the user one after .. so it seems like it's pretty .. fragile :)

I that exact race condition too. Can you try to: * Install via the Homebrew instructions * Restart your computer * Login to your computer * Run `ssh-add -l; launchctl list | grep ssh` and paste the output here. * Log out and login again * Run `ssh-add -l; launchctl list | grep ssh` and paste the output here. @theseal on the other hand has the exact opposite problem: it works after boot but not if he logs out and in again = D Race conditions are funny things eh!? At least this one is sort of deterministic… I sadly have no idea how to solve this except loging into single user mode, remove SIP, disable the com.openssh.ssh-agent plist, enable SIP, reboot and live happily ever after...

[lukaskuzmiak]

so I got a new computer and had to set this up again. Same exact thing as before. The system plist is loaded first, I tried MANY times and did not see the homebrew one loaded as first at all.

But I found a little easier way to edit/disable the system one. When you reboot into recovery you can launch Disk Utility, mount the hard drive, exit disk utility, Tools -> Terminal and do whatever you need without removing SIP, rebooting, editing, rebooting, enabling SIP, rebooting .. so save a bunch of time rebooting :-)

Anybody else can share their experience on Mojave?

Do you think it is better to disable the system one or edit it to contain the SSH_ASKPASS + DISPLAY?

[simmel]

@lukaskuzmiak But logging out and logging in still makes it use the homebrew one?

[lukaskuzmiak]

@simmel did not see that happen in my case, not sure what could be influencing it.

[theseal]

What happens if you run brew services start ssh-askpass before you try to add the key(s) to the agent after a reboot?

Edit:
Seems like brew services changed behaviour. The command you probably would like to run is launchctl start homebrew.mxcl.ssh-askpass.

[lukaskuzmiak]

Sorry, I don't have a system to test it with at the moment, I'll try to get back to this but if anybody can give it a shot or report a working Mojave setup that'd help.

EDIT: my co-worker reported that the system plist is not there unless you have Xcode installed (or maybe just Xcode command line tools), no way to verify this atm, feel free to post if you see a correlation there.

[ttimasdf]

TL;DR:
Solution:

brew services stop ssh-askpass
sudo cp /usr/local/opt/ssh-askpass/homebrew.mxcl.ssh-askpass.plist /Library/LaunchAgents

Then reboots.

---

The main point is, how to set the correct SSH_AUTH_SOCK and SSH_ASKPASS globally. In Mojave from what i can tell, the brew services cannot provide the functionality that this utility needs.

If I run brew services start ssh-askpass. The launchd plist is put into ~/Library/LaunchAgents. I guess agents under this directory is launched after Finder or whatever GUI things. So the environment variables is NOT injected properly.

If I run sudo brew services start ssh-askpass instead, the plist is copied to /Library/LaunchDaemons which launched as root instead of one instance per user. It placed the environment variable correctly but that's kinda environment pollution because it declared an SSH_AUTH_SOCK owned by root that nobody could ever access.

Manually copy the plist into /Library/LaunchAgents works like a charm and do not interfere with system com.openssh.ssh-agent. I guess agents under /L/L is launched after /S/L/L so it override the environment variable set by system ssh-agent.

Run launchctl print gui/$(id -u)/com.openssh.ssh-agent you could see output like this:
(sudo unneeded)

com.openssh.ssh-agent = {
        active count = 0
        path = /System/Library/LaunchAgents/com.openssh.ssh-agent.plist
        state = waiting
...
        inherited environment = {
                SSH_AUTH_SOCK => /private/tmp/com.apple.launchd.37tpUsjwaE/Listeners
                Apple_PubSub_Socket_Render => /private/tmp/com.apple.launchd.8QwTYRDevb/Render
                SSH_AUTH_SOCK => /private/tmp/com.apple.launchd.xQvJEJNVSa/Listeners
        }

Actually there are two agents waiting and the one which is accessed by client is activated by launchd. Whichever one is okay because the two are identical. In practice, the ssh-agent launched by us is actually running.

If you prefer that only one ssh-agent running at a time. You could optionally disable the system ssh-agent for current user.

# sudo unneeded
launchctl disable gui/$(id -u)/com.openssh.ssh-agent

[theseal]

Thanks for your post Tim!
It is hard for me to test out your solution since everything already works just like it should on my Mojave machine. Do you have any idea why that is? Race condition?

Is it any disadvantages to disable the system ssh-agent for the current user? Would it work to just to do so in order to fix this issue? It should reduced the fix to just one command which isn't dependent on sudo. Any thoughts about that?

Do you still have this issue @lukaskuzmiak and care to test it out?

[ttimasdf]

I thought it was, but placing the plist in ~/Library/LaunchAgents just not work with or without system agent. After disabled system agent, KeepassXC will not detect ssh-agent started by us, either.

As your setup works just fine I could make another assumption. My machines (a mac and a hackintosh) both installed ssh-askpass after upgraded to Mojave (from 10.13.x directly to 10.14.1), surely you installed before. 😄 Could that make any difference? That Mojave behave differently with the precedence.

[stromnet]

For what it's worth, yesterday I got a new clean Mojave installation, and for me it was sufficient to copy the file to ~/Library/LaunchAgents/ (and of course the script to /u/l/b). So far it has worked as expected.

One caveat:

The first time I ran, I got a prompt from the new Security and Privacy system, asking if I would allow ssh-askpass binary to access SystemEvents (or whatever it was, don't have that computer in front of me ATM). If I pressed No, it would not allow it to use AppleScript/prompts, and ssh would say something like agent refused operation. And it only asked the first time..
Eventually I figured out that I reset that decision with the tccutil reset SystemEventOrSometihngNotSure, and then it asked again. Pressing Yes this time, it has worked since.

[simmel]

I clean installed Mojave and I still get the race condition described above = S I mean, all race conditions are weird but intermittent race conditions are way worse = )

[simmel]

On Mon, 2019-01-07 at 19:37:22 -0800, Tim Zhang wrote: I thought it was, but placing the plist in `~/Library/LaunchAgents` just not work with or without system agent. After disabled system agent, KeepassXC will not detect ssh-agent started by us, either.

But removing it from ~/Library/LaunchAgents, putting it in /Library/LaunchAgents and disabling the build in ssh-agent still works for you now? If so, great! Let's just hope that OP get a chance to test this workaround. I'm AFK sadly.

As your setup works just fine I could make another assumption. My machines (a mac and a hackintosh) both installed ssh-askpass *after* upgraded to Mojave (10.14.1), surely you installed *before*. 😄 Could that make any difference? That Mojave behave differently with the precedence.

Our co-worker which also clean installed and installed after has no problem either...

[mikeziri]

TL;DR:
Solution:

brew services stop ssh-askpass
sudo cp /usr/local/opt/ssh-askpass/homebrew.mxcl.ssh-askpass.plist /Library/LaunchAgents

Then reboots.

The main point is, how to set the correct SSH_AUTH_SOCK and SSH_ASKPASS globally. In Mojave from what i can tell, the brew services cannot provide the functionality that this utility needs.

If I run brew services start ssh-askpass. The launchd plist is put into ~/Library/LaunchAgents. I guess agents under this directory is launched after Finder or whatever GUI things. So the environment variables is NOT injected properly.

If I run sudo brew services start ssh-askpass instead, the plist is copied to /Library/LaunchDaemons which launched as root instead of one instance per user. It placed the environment variable correctly but that's kinda environment pollution because it declared an SSH_AUTH_SOCK owned by root that nobody could ever access.

Manually copy the plist into /Library/LaunchAgents works like a charm and do not interfere with system com.openssh.ssh-agent. I guess agents under /L/L is launched after /S/L/L so it override the environment variable set by system ssh-agent.

Run launchctl print gui/$(id -u)/com.openssh.ssh-agent you could see output like this:
(sudo unneeded)

com.openssh.ssh-agent = {
        active count = 0
        path = /System/Library/LaunchAgents/com.openssh.ssh-agent.plist
        state = waiting
...
        inherited environment = {
                SSH_AUTH_SOCK => /private/tmp/com.apple.launchd.37tpUsjwaE/Listeners
                Apple_PubSub_Socket_Render => /private/tmp/com.apple.launchd.8QwTYRDevb/Render
                SSH_AUTH_SOCK => /private/tmp/com.apple.launchd.xQvJEJNVSa/Listeners
        }

Actually there are two agents waiting and the one which is accessed by client is activated by launchd. Whichever one is okay because the two are identical. In practice, the ssh-agent launched by us is actually running.

If you prefer that only one ssh-agent running at a time. You could optionally disable the system ssh-agent for current user.

# sudo unneeded
launchctl disable gui/$(id -u)/com.openssh.ssh-agent

thank you!

[simmel]

launchctl disable gui/$(id -u)/com.openssh.ssh-agent

I can confirm that this works (for me). Cool! Finally something that works deterministically!

I thought it was, but placing the plist in ~/Library/LaunchAgents just not work with or without system agent. After disabled system agent, KeepassXC will not detect ssh-agent started by us, either.

Did you get this working @ttimasdf?

Sounds like KeepassXC just uses the SSH_AUTH_SOCK?
When KeepassXC is started, is it's SSH_AUTH_SOCK environment variable (check ps auxwwe | grep '[K]eepassXC') the same as what you have in your shell?

[simmel]

Haha, it didn't work after reboot...

[arunkant]

brew services stop ssh-askpass
sudo cp /usr/local/opt/ssh-askpass/homebrew.mxcl.ssh-askpass.plist /Library/LaunchAgents

worked for me

[simmel]

Did you have an intermittent problem or always failure? Does it still work after reboot?

…

On Fri, 2019-07-05 at 20:37:38 -0700, arunkant wrote: > brew services stop ssh-askpass > sudo cp /usr/local/opt/ssh-askpass/homebrew.mxcl.ssh-askpass.plist /Library/LaunchAgents worked for me -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #23 (comment)

[arunkant]

It was failing every time, although I only tested it 4-5 times.
Had zero problems after changes, even after reboot.

[petiepooo]

Since this issue is still open, I'll post an update. It appears that a recent security update to Mojave "fixed" the race condition. And by fixed, I mean broke it for me...
I had needed to logout and log back in since installing Mojave in order to get askpass to work for me. However, a couple of days ago, even that stopped working.
Checking the environment for launchctl, it appeared the DISPLAY variable was already set, so I just ran

launchctl setenv SSH_ASKPASS /usr/local/bin/ssh-askpass

then restarted the ssh-agent and everything worked again. I expect it will work without a logoff/login now too.

[simmel]

Since this issue is still open, I'll post an update. It appears that a recent security update to Mojave "fixed" the race condition. And by fixed, I mean broke it for me...
I had needed to logout and log back in since installing Mojave in order to get askpass to work for me. However, a couple of days ago, even that stopped working.
Checking the environment for launchctl, it appeared the DISPLAY variable was already set, so I just ran

launchctl setenv SSH_ASKPASS /usr/local/bin/ssh-askpass

then restarted the ssh-agent and everything worked again. I expect it will work without a logoff/login now too.

I can confirm that it does work without logoff/login.

Thanks Apple for fixing it and a very big Thank you! to @petiepooo for your persistance in trying to solve the problem!