Cannot use SSH URLs for source/release repositories in rosdistro repo

[jjekircp]

Hi all,

After noticing I couldn't use SSH URLs in my distribution.yaml, I started looking into what I'd have to change to get SSH key support inside the Docker container for the rosdistro cache job.

I referenced moby/moby#6396 and I succeeded in forwarding the SSH agent into the container for the rosdistro cache job, but I ran into trouble with generate_all_jobs.py as I added the build-wrapper_ssh-agent snippet into templates/misc/rosdistro_cache_job.xml.em and it needs a credential ID.

That is, in scripts/misc/generate_rosdistro_cache_job.py lines 37-60 in get_job_config, I need to populate the job_data dictionary with all the values needed to expand the template, and I'm not sure where to get the credential ID from.

The release build configuration in ros_buildfarm_config has an "upload_credential_id", but in the general case that may not be the credential the user wishes to use to fetch private git repositories. Perhaps there is a need for another credential_id in the index.yaml in that repository?

I'd appreciate your feedback on how that should be implemented.

For reference, the work I've done so far is on the ssh_git_clone branch of my fork.

[tfoote]

To do this you'd probably want to setup a new set of credentials on the master.

It's configured here:
https://github.com/ros-infrastructure/buildfarm_deployment_config/blob/master/master/common.yaml#L13

With the template here: https://github.com/ros-infrastructure/buildfarm_deployment/blob/master/master/jenkins_files/templates/credentials.xml.erb#L11

Configured by puppet here: https://github.com/ros-infrastructure/buildfarm_deployment/blob/master/master/manifests/site.pp#L457

You would then want to embed the matching credential ID into your config passed to the ssh-agent snippet in the buildfarm config.

Our implementation did not support arbitrary credential adding but I think you could extend it to do that relatively easily.

I'll note that deployment to the buildservers is not the reason we don't support ssh credentials. As you've determined that is a tractable problem. The reason we don't allow ssh urls is that they require you to have your ssh keys registered with github and/or bitbucket to clone onto end users machines. Thus if they use the rosinstall generator to compile from source they would have to have installed ssh keys onto every machine which are registered with all the upstream sources, otherwise they cannot get access even though the source is open and freely available.

In the case of a proprietary sources though it's a little different and the ssh key approach is probably easier than setting up the non-interactive https authentication. (Though that is also possible and might be something to consider too)

[jjekircp]

I'm pretty close to putting up a PR for this, but there's one problem I've run into that I haven't been able to find a nice way to solve.
In buildfarm_deployment I've added a template to fill out a known_hosts file, along with a corresponding option in buildfarm_deployment_config.

The config options look like this:

ssh_known_hosts:
    'github.com': |
        |1|/F/a+D+AA/y+qf7+IMSwXbvfFZo=|Pygbd2OeNdWzbgAyZK/kwEet9u0= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
    'bitbucket.org': |
        |1|VoTP5i1zOk28A+ELJ0XpcMdpiBc=|Y61MET377AK92/9wJzCZhQMoGmw= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAubiN81eDcafrgMeLzaFPsw2kNvEcqTKl/VqLat/MaB33pZy0y3rJZtnqwR2qOOvbwKZYKiEO1O6VqNEBxKvJJelCq0dTXWT5pbO2gDXC6h6QDXCaHo6pOHGPUy+YBaGQRGuSusMEASYiWunYN0vCAI8QaXnWMXNMdFP3jHAJH0eDsoiGnLPBlBp4TNm6rYI74nMzgz3B9IikW4WVK+dc8KZJZWYjAuORU3jc1c/NPskD2ASinf8v3xnfXeukU0sJ5N6m5E8VLjObPEO+mN2t/FZTMZLiFqPWc/ALSqnMnnhwrNi2rbfg/rd/IpL8Le3pSBne8+seeFVBoGqzHM9yXw==

What I was trying to do was write a template that would leave the known_hosts file blank if no ssh_known_hosts variable was set in slave/common.yaml. However, no matter what I do the scope.function_hiera(["ssh_known_hosts"]) call I make inside the template gives an error when puppet runs if the ssh_known_hosts variable isn't set.

From the research I did so far (I don't really have any experience with Puppet or Ruby; I'm mostly going by what I see done for other things in the buildfarm_deployment repo) it doesn't look like there's a way to specify a fallback value for a hiera variable unless one defines the variable in the class then does the hiera lookup there (as opposed to doing it directly in the template like it's done here).

I'd appreciate if you could provide a hint as to how this should be done. It might also be possible to leave the default config string above present in common.yaml, then every copy of buildfarm_deployment_config would have a valid value for that variable. I was trying to make it optional for neatness as I expect that some significant number of users won't care about having that option.

For background, the reason that is there is so that for devel and doc jobs, which use the Jenkins SCM plugin to clone, the Jenkins user needs to have all servers that'll be used for Git over SSH in its known_hosts or else it will fail as it does not recognize the hostkey.