Best practices for pulling from private git repo?

[jessep]

Hi, just wondering if you have thoughts on best way to get this to work if you're pulling from a private repo. My initial thought is:

1. create a non-passphased ssh key pair just for github
2. add it to the github admin
3. within roles/web/tasks/setup_git_repo.yml have ansible add private key to the user account on the target machine.

Does that sound like an ok plan?

[jcalazan]

Hey Jesse,

Yup, that's pretty much what I do for my private repos.

I have something like this:

- name: Create the SSH public key file
  copy: content="{{ git_ssh_pub_key }}"
        dest=/root/.ssh/id_rsa.pub
        mode=0644

- name: Create the SSH private key file
  copy: content="{{ git_ssh_priv_key }}"
        dest=/root/.ssh/id_rsa
        mode=0600

- name: Setup the Git repo
  git: repo={{ git_repo }} dest={{ project_path }} accept_hostkey=yes
  when: setup_git_repo

You'd probably also want to encrypt the file containing the keys with Ansible Vault if you're committing them to the repo.

Another option is to use a private base image with the user and Git keys already created.

[jessep]

Awesome thanks. Would you mind if I submitted a pull request to make
setting this up easier for others? Seems like something many people using a
real project would need.
On Sun, Oct 26, 2014 at 9:14 PM JC notifications@github.com wrote:

Hey Jesse,

Yup, that's pretty much what I do for my private repos.

I have something like this:

• name: Create the SSH public key file
  copy: content="{{ git_ssh_pub_key }}"
  dest=/root/.ssh/id_rsa.pub
  mode=0644
• name: Create the SSH private key file
  copy: content="{{ git_ssh_priv_key }}"
  dest=/root/.ssh/id_rsa
  mode=0600
• name: Setup the Git repo
  git: repo={{ git_repo }} dest={{ project_path }} accept_hostkey=yes
  when: setup_git_repo

You'd probably also want to encrypt the file containing the keys with
Ansible Vault if you're committing them to the repo.

Another option is to use a private base image with the user and Git keys
already created.

—
Reply to this email directly or view it on GitHub
#11 (comment)
.

[jcalazan]

That would be great! Perhaps add a when: condition to only run those tasks if the git key vars are defined.

Thanks!

[jessep]

Cool, sounds good. I currently am using templates to store the key file,
seems more natural to me than a var. Is there a rationale for using vars
for the ssl certs and git keys instead of templates?

On Mon, Oct 27, 2014 at 12:05 PM, JC notifications@github.com wrote:

That would be great! Perhaps add a when: condition to only run those
tasks if the git key vars are defined.

Thanks!

—
Reply to this email directly or view it on GitHub
#11 (comment)
.

[jcalazan]

Hmm...not sure what you mean by templates. Are they basically just separate jinja files that get copied to the server (like ssl_cert.j2, ssl_key.j2, id_rsa.j2, id_rsa_pub.j2)?

I like putting them in the vars_file so I only have to deal with one file that has all my secret stuff which I then encrypt with Ansible Vault. Not sure if that's considered best practice but I like it for its simplicity.

[kulbir]

@jcalazan Hi tried your solution. Private and Public keys are copied in vagrant machine. But it does not work. I am getting permission error.

- name: Create the SSH public key file
  copy: content="{{ git_ssh_pub_key }}"
        dest=/root/.ssh/id_rsa.pub
        mode=0644

- name: Create the SSH private key file
  copy: content="{{ git_ssh_priv_key }}"
        dest=/root/.ssh/id_rsa
        mode=0600

- name: Setup the Git repo
  git: repo={{ git_repo }} dest={{ project_path }} accept_hostkey=yes
  when: setup_git_repo

error

stderr: Permission denied (publickey).
fatal: The remote end hung up unexpectedly

msg: Permission denied (publickey).
fatal: The remote end hung up unexpectedly

FATAL: all hosts have already failed -- aborting

Any idea? Thanks

[alfondotnet]

I had to do this before in order for the file to be correctly created.

---

- name: Creates directory
  file: path=/root/ state=directory

- name: Creates directory
file: path=/root/.ssh state=directory

[jcalazan]

@kulbir Ah, I haven't tried this with Vagrant, I've only done it with Digital Ocean which creates a root user by default and already has a .ssh folder.

@alfonsoperez's solution will probably work if that's the case. If not, add the -vvvv at the end of the ansible command to get more info (and if you can provide the name of the task where it failed that will help also). I'm surprised you didn't get an error while creating the files.

[kulbir]

Hi @jcalazan

Thanks for your response.

I am finally able to clone the private repo at vagrant's home folder (/home/vagrant/). but not at /webapps/.

Code

---

- name: Create the Github SSH public key file
  copy: src="/Users/sandhu/.ssh/id_rsa.pub"
        dest="/home/vagrant/.ssh/id_rsa_github.pub"
        mode=0644
        owner=vagrant
        group=vagrant

- name: Create the Github SSH private key file
  copy: src="/Users/sandhu/.ssh/id_rsa"
        dest="/home/vagrant/.ssh/id_rsa_github"
        mode=0600
        owner=vagrant
        group=vagrant

- name: Setup the Git repo
  sudo: no
  git: repo={{ git_repo }}
       version={{ git_branch }}
       dest={{ project_path }}
       accept_hostkey=yes
       key_file="/home/vagrant/.ssh/id_rsa_github"

  when: setup_git_repo is defined and setup_git_repo
  tags: git

But I'm not able to clone directly to \webapps\etc folder.

1. When I try to clone directly to \webapps\etc folder with sudo:yes. Then git module hanged. Here are the verbose's output.

TASK: [web | Setup the Git repo] ********************************************** 
<127.0.0.1> ESTABLISH CONNECTION FOR USER: vagrant
<127.0.0.1> REMOTE_MODULE git repo= version=master dest=/webapps/climate_assets/assets accept_hostkey=yes key_file="/home/vagrant/.ssh/id_rsa_github"
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/sandhu/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/sandhu/projects/ansible/.vagrant/machines/default/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 127.0.0.1 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1433706858.94-280175940993855 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1433706858.94-280175940993855 && echo $HOME/.ansible/tmp/ansible-tmp-1433706858.94-280175940993855'
<127.0.0.1> PUT /var/folders/5t/pcld8jgs7k97mwm182lfw6hh0000gn/T/tmpZ7HDrE TO /home/vagrant/.ansible/tmp/ansible-tmp-1433706858.94-280175940993855/git
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/sandhu/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/sandhu/projects/ansible/.vagrant/machines/default/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=10 127.0.0.1 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=hifaazzegckyjaykjzwljucpsyxztyht] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-hifaazzegckyjaykjzwljucpsyxztyht; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1433706858.94-280175940993855/git; rm -rf /home/vagrant/.ansible/tmp/ansible-tmp-1433706858.94-280175940993855/ >/dev/null 2>&1'"'"''

2. When I try to clone directly to /webapps/etc folder with sudo:no. then I get permission error. Yeah. I need sudo access to clone/copy the contents to /webapps/

failed: [default] => {"cmd": "/usr/bin/git clone --origin origin --branch master '' /webapps/climate_assets/assets", "failed": true, "rc": 128}
stderr: fatal: could not create work tree dir '/webapps/climate_assets/assets'.: Permission denied

msg: fatal: could not create work tree dir '/webapps/climate_assets/assets'.: Permission denied

FATAL: all hosts have already failed -- aborting

Should I copy github code each time to /webapps/? Can't I clone directly to /webapps/ folder?
or Am I missing something?

So what do you suggest?

Thanks,
Kulbir

[jcalazan]

Hmm...is your Git key password protected by any chance? If it hung, I'm thinking it might be asking for a password. You can try creating another set of keys without a password and use that.

The /webapps owner is set to the sudo user by default. You can add a task here to create the /webapps directory separately (the very beginning, before creating the virtualenv folder) with its own permissions if you like, or simply specify the owner in the first task when initially creating the virtualenv folder (since the path is something like /webapps/app_name, the /webapps path got created automatically, which defaults to the sudo user as the owner):

https://github.com/jcalazan/ansible-django-stack/blob/master/roles/web/tasks/setup_virtualenv.yml

[kulbir]

Hi @jcalazan

Thank you very much for your response. Yeah, my git keys was password protected. I created another set of keys without a password. And it worked. :-)

Working git code.

- name: Create the Github SSH public key file
  copy: src="/Users/sandhu/.ssh/id_rsa.pub"
        dest="/root/.ssh/id_rsa.pub"
        mode=0644

- name: Create the Github SSH private key file
  copy: src="/Users/sandhu/.ssh/id_rsa"
        dest="/root/.ssh/id_rsa"
        mode=0600

- name: Setup the Git repo
  git: repo={{ git_repo }}
       version={{ git_branch }}
       dest={{ project_path }}
       accept_hostkey=True
       key_file="/root/.ssh/id_rsa"

Thank You.

[jcalazan]

Np, glad that worked! :)

[phillpafford]

so are you just copying the public/private keys to your vagrant?

Im trying to do something similar but generate the keys on the vagrant itself and push the public key to my private repo via API for read-only access

[jcalazan]

I have a task to copy over the keys (encrypted with Ansible Vault before committing to my repo) to the destination server.

If you're using Vagrant just for development, simplest way is probably just share your local Git directory to the VM.

There's also something called "SSH Agent forwarding" that lets you use your local SSH keys instead of putting them on the server, but haven't played around with it yet: https://developer.github.com/guides/using-ssh-agent-forwarding/

[kjagiello]

SSH agent forwarding is the way to go. That way you won't need any extra SSH key and can simply use the one on your machine without copying it around.

[scuerda]

I'm was having issues similar to what @kulbir was experiencing.

I generated a key that I associated with bitbucket for accessing a private repo. I confirmed that it is not passphrase protected. on the machine where I created it. Yet, when I create it on my vagrant box, it hangs up when expecting a passphrase. If I ssh in to the vagrant box and try to manually clone the repo in question, I am prompted for a passphrase.

If I switch to an approach where I pass in the actual keys, vs. setting the value as variables in the base.yml file, everything works. Any ideas as to why that might be?

[pogorelov-ss]

I have config with SSH agent forwarding,
https://gist.github.com/pogorelov-ss/41893e17c7c4776d4d57

[snwflake]

I don't quite understand the discussion.
Vagrant comes with an option to enable agent forwarding. config.ssh.forward_agent = true
The only thing left todo is enable agent forwarding in your ssh conf:

Host <inventory_hostname>
ForwardAgent yes

[YPCrumble]

@snwflake did you try that and did it work?

I suspect if you try it you'll find you need other changes I had in #67

[snwflake]

I'm using this approach in my current playbook. The only thing you have to make sure, is that you set become: no to the task when cloning the repository and logon to the machine with vagrant. If you become root, the SSH_AUTH_SOCK env var is not accessible by root and the ForwardAgent is lost.

[YPCrumble]

@snwflake thanks! I'm curious how you were able to overcome a couple obstacles I had to adjust the playbook for:

How did you add your repository URL to the known_hosts file or did you have a workaround for that?

Also, did you have a workaround for setting file permissions such that the non-root user can clone the repo? Currently the file permissions are set after the repo is cloned so I had to move those up in my repo.

See https://github.com/jcalazan/ansible-django-stack/pull/67/files for what I had done.

[snwflake]

As long as it is a dedicated dev machine, running on localhost, I wouldn't mind simply setting accept_hostkey. In a prod env, I would create a template for the known_hosts and inject it accordingly.
I don't quite understand what you were trying to accomplish here.
You touch the known_hosts file and then check if github.com has already a key inside. I mean, that doesn't really make sense.

As for the file permissions, I don't have any problems with that, since I chown my venv-dir as soon as everything else is set up. Setting up git-repo and user space only requires vagrants permissions therefore it's set correctly. Depending on your env (dev, stage, prod) I would secure all file permissions at the end, but I don't see how that is beneficial in a dev env.

Edit: fixed english

[StuartMacKay]

I have a private git repository (hosted on WebFaction). I was able to able to get SSH forwarding working with the following setup:

1. Enable SSH forwarding in the Vagrant VM.

# Vagrantfile
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "ubuntu/xenial64"
  config.ssh.forward_agent = true
  ...

2. Allow any user on the target to use the ssh forwarding socket.

In env_vars/base.yml, add the flag, git_repo_private. This is just a convenience so the task added in the next step does not run if the repository is public. I don't think it has any value other than being cosmetic.

# env_vars/base.yml 
---

git_repo: ...
git_repo_private: true
...

In roles/web/tasks/setup_git_repo.yml, add a new task, "Add ssh agent line to sudoers" BEFORE the "Setup the Git repo" task. Note the addition of the "environment" field to set TMPDIR in the git command.

# roles/web/tasks/setup_git_repo.yml

# Make sure the sudoers file preserves the ability to use ssh forwarding.
# That way we don't need to store a private key on the server to get
# access to the git repository. Don't forget to add the key used by the
# git repository to your ssh-agent using ssh-add on the machine where you
# run the playbooks.
#
# https://stackoverflow.com/questions/24124140/ssh-agent-forwarding-with-ansible

- name: Add ssh agent line to sudoers
  lineinfile:
    dest: /etc/sudoers
    state: present
    regexp: SSH_AUTH_SOCK
    line: Defaults env_keep += "SSH_AUTH_SOCK"
  when: git_repo_private

# The git module calls python's tempfile.mkstemp() which uses the TMPDIR
# environment variable. However this is set to /tmp which is mounted as
# noexec. As a result the git command will fail. The solution is to set
# TMPDIR to point to some other suitable location. Here we use /var/tmp
# but any suitable location will do.
#
# https://github.com/ansible/ansible/issues/30064
# https://docs.python.org/dev/library/tempfile.html?highlight=mkstemp#tempfile.tempdir

- name: Setup the Git repo
  environment:
      TMPDIR: "/var/tmp"
  git: repo={{ git_repo }}
       version={{ git_branch }}
       dest={{ project_path }}
       accept_hostkey=yes
  when: setup_git_repo is defined and setup_git_repo
  notify: restart application
  tags: git

I am just getting started with this project. My experience so far is this works when provisioning the Vagrant VM. The panel to unlock the key pops up on the first checkout and not in subsequent ones. I have not tried this on a droplet yet.

It took quite a bit of fiddling to get the VM to first work so what I don't know is whether 'vagrant destroy' really cleans out everything and nothing is left lying around that affects the changes above.

[YPCrumble]

Fixed via #91