SSH host connection assumption

[alexrussell]

As per the discussion on this thread, this is an issue just putting the things learnt on the radar here.

The general thing is that if using a 'Remote URL'-type repository and that host has never been connected to by the user that runs the cron job (which may not be the same as the user that the webserver runs as, just to be extra confusing) then any build will likely fail due to the "unable to verify the authenticity of this host" message (and expectation of input - use needs to type "yes" at the prompt) that comes from SSH connecting to a host that is not listed in ~/.ssh/known_hosts

I originally suggested that at the least the installation insturction or the "new/edit project" page should contain a warning that you'll need to go in and connect to the host manually from the user, but to be honest that's unacceptable and I found a better way.

The better way would be to, upon adding (or editing if the clone URL changes) a project that is a 'Remote URL'-type project, the webserver should quickly attempt to SSH to the host to run an arbitrary command using the -oScrictHostKeyChecking=no command line flag. This should connect to the host, ignore the "add this to the list?" question and actually silently add the host to the list. Some/most git servers don't actually allow an interactive TTY (so the command may fail), but simply having a connection should be enough to achieve what we want this.

It may also be worth saying in the installation instructions that the webserver and the cron job must run as the same user. This is because if the website initially does this connection the host will go into that user's ~/.ssh/known_hosts file, but when the cron job comes to run the builds, PHP runs as that user and so SSH will check their ~/.ssh/known_hosts file. Maybe this isn't a big deal, but the installation instructions are fairly loose as-is, and I worry that some people will run the website as the standard www-data user (default with Apache + PHP and Nginx + PHP-FPM I think) but the cron job as their local user. I haven't because I didn't want to have to deal with possible file permission issues (and didn't want to chmod 777 all the things!) so my setup runs both as www-data.

[gabriel403]

Something like this could always be done by the cron
echo -e "Host github.com\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config
or perhaps the solution above should be done by the cron? I don't think it needs to be done by the webserver just incase the cron is being run by a different user

[alexrussell]

@gabriel403 I agree in general, and my ideal solution would have been to involve a queue (we don't want the server doing this every minute, just the once will be fine) that the cron job also processes as well as the builds. This could certainly be a general improvement to the build system as I think it currently just scans the build table for stuff not yet started. That's fine, but put the builds in a queue and suddenly the queue can be used for other stuff.

And yeah, either attempting a connection as in my method or just making sure that ssh doesn't try to ask the user to accept the key as in your method is fine. Either way, the first time ssh connects to the remote host the user won't be asked and then the key will go in ~/.ssh/known_hosts and every time after it doesn't care about that unless the key changes, and using either method, if the key changes ssh will whine about it and fail the build. I think the only way around that is changing the known_hosts file to /dev/null on the command line, but I dunno how well that works with git. As I said above, there are some commands (like rsync) where you can tell them how to interact with SSH, but some (maybe git?) don't allow that.

[dancryer]

After trying a few times, I've come to the conclusion we're probably not going to be able to fix this. Most people can get around it by either turning off host key checking, or connecting manually before trying to do so via PHPCI.

It's a pain, but git doesn't seem to provide any alternative.

[gabriel403]

I wonder if we can do anything like puppet does here https://github.com/puppetlabs/puppetlabs-vcsrepo/blob/master/lib/puppet/provider/vcsrepo/git.rb#L306

[gabriel403]

I use this ruby gem and it does something similar https://github.com/martinemde/git-ssh-wrapper/blob/master/lib/git-ssh-wrapper.rb#L68

[dancryer]

Would you be interested in having a go at making that work? It looks like it could also remove the need to use ssh-agent. :)

[dancryer]

Nevermind, seems it was easier than I thought. This should be working now. :)

[gabriel403]

👍

[alexrussell]

Woohoo! Good work both in finding a nice simple solution.