Ansible based setup script.

[glogiotatidis]

This is a first try to automate install of development environment using Ansible. This is neat because:

• ansible uses modules to do things instead of custom bash / python scripts and commands.
• the same ansible script can be used on your local machine, docker container or VM.
• the script runs all our unit tests in the end, thus verifies that our script is correct. We can have a CI job to run this script (i.e. setup an enviroment and run all our tests) to verify that our installation instructions are correct.
• We can have different scripts for different hosts systems (i.e. fedora) fairly easy while keeping it DRY

I submit this pull request for comments @akatsoulas @johngian

[axilleas]

I tried it on Ubuntu 14.04 and some modifications had to be made.

• Add sudo: yes where needed. Most parts should be ran with our home user.
• Add python-pil as it was missing from deps.
• Rename package node to nodejs.
• Node provides /usr/bin/nodejs and somewhere in the code node is expected, so I created a symlink to /usr/local/bin/node. Probably this should be fixed in the code.
• Create mysql user mozillians and grant perms on mozillians db.
• Check and copy django settings/local.py
• Migrate db.
• Create elasticsearch_version variable because it pointed to wrong path and the Run ElasticSearch task was failing.

I couldn't make the tests to run. They need a test_mozillians db and while it is created with the Create mysql database task, I cannot grant it privileges to mozillians user with the Create mysql user task. It's either that or it needs a password to run. So far all db users are passwordless.
I temporarily removed the test task from the playbook.

Future enhancements

• Break it to roles
• Support more distros

Steps to run

Place ansible.yml in mozillians/.

apt-get install ansible
git clone --recursive git@github.com:YOUR_USERNAME/mozillians.git mozillians
cd mozillians/
echo 127.0.0.1 > hosts
ansible-playbook -i hosts ansible.yml --ask-sudo-pass
./manage.py runserver

ansible.yml

---
- hosts: 127.0.0.1
  connection: local

  tasks:
    - name: Install needed packages
      sudo: yes
      apt: name={{ item }} state=present update_cache=yes cache_valid_time=300
      with_items:
        - python-virtualenv
        - python-pil
        - build-essential
        - python-dev
        - gettext
        - libjpeg-turbo8-dev
        - mysql-client
        - mysql-server
        - libmysqlclient-dev
        - subversion
        - nodejs
        - npm
        - libxslt1.1
        - libxml2
        - libxml2-dev
        - libxslt1-dev
        - openjdk-7-jre
        - python-mysqldb

      tags:
        - setup
        - packages

    - name: Create nodejs symlink
      sudo: yes
      file: src=/usr/bin/nodejs dest=/usr/local/bin/node state=link

    - name: Install compiled python packages
      pip:
        requirements={{ path }}/requirements/compiled.txt
        virtualenv={{ virtualenv_path }}
      tags:
        - setup
        - packages

    - name: Install development python packages
      pip:
        requirements={{ path }}/requirements/dev.txt
        virtualenv={{ virtualenv_path }}
      tags:
        - setup
        - packages

    - name: Install npm packages
      sudo: yes
      npm: name=less global=yes
      tags:
        - setup
        - packages

    - name: Create mysql database
      sudo: yes
      mysql_db: name={{ item }} state=present
      with_items:
        - mozillians
      tags:
        - setup
        - config
        - database

    - name: Create mysql user
      sudo: yes
      mysql_user: name=mozillians password= priv={{ item }}.*:ALL state=present
      with_items:
        - mozillians
      tags:
        - setup
        - config
        - database

    - name: Check if settings/local.py exists
      stat: path={{ django_path }}/mozillians/settings/local.py
      register: settingspy_stat

    - name: Copy django settings
      copy: src={{ path }}/mozillians/settings/local.py-devdist dest={{ path }}/mozillians/settings/local.py
      when: not settingspy_stat.stat.exists
      tags:
        - setup

    - name: Syncdb
      django_manage:
        command=syncdb
        app_path={{ django_path }}
        virtualenv={{ virtualenv_path }}
      tags:
        - setup
        - syncdb
        - database

    - name: Migrate database
      django_manage:
        command=migrate
        app_path={{ django_path }}
        virtualenv={{ virtualenv_path }}
      tags:
        - setup
        - migrate
        - database

    - name: Check if ElasticSearch exists
      stat: path={{ elasticsearch_path }}
      register: elasticsearch_stat
      tags:
        - setup
        - elasticsearch

    - name: Download ElasticSearch
      get_url:
        url={{ elasticsearch_url }}
        dest={{ elasticsearch_tmp }}
      when: not elasticsearch_stat.stat.exists
      tags:
        - setup
        - elasticsearch

    - name: Extract ElasticSearch
      unarchive: src={{ elasticsearch_tmp }} dest={{ path }} copy=no
      when: not elasticsearch_stat.stat.exists
      tags:
        - setup
        - elasticsearch

    - name: Delete temp files
      command: rm {{ elasticsearch_tmp }}
      when: not elasticsearch_stat.stat.exists
      tags:
        - setup
        - elasticsearch

    - name: Run ElasticSearch
      command: "{{ elasticsearch_path }}/bin/elasticsearch"
      register: run_elasticsearch
      tags:
        - run

    - name: Wait for ElasticSearch to come up
      wait_for: port=9200 delay=5
      when: run_elasticsearch

  vars:
    - path: "{{ lookup('env', 'PWD') }}"
    - django_path: "{{ path }}"
    - virtualenv_path: "{{ path }}/.venv"
    - elasticsearch_version: "0.90.10"
    - elasticsearch_path: "{{ path }}/elasticsearch-{{ elasticsearch_version }}"
    - elasticsearch_tmp: /tmp/elasticsearch.tar.gz
    - elasticsearch_url:
        https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.10.tar.gz

[glogiotatidis]

@axilleas thanks for trying this out man! This is by no means complete. A few of the things you mention (like node vs nodejs) can be fixed by supporting different distros as you mention and others just need fixing.

Do you think that having ansible playbooks for development environment setup (especially for contributors) could be the way forward?

One thing I like about them is that you can setup a CI job to test the actual installation, thus I have the django tests as the very last step of the playbook. If the tests pass, installation was successful.

I'd love to hear your thoughts on this.

[m8ttyB]

Just curious, why Ansible and not something that we already use and is supported [like Puppet]?

[johngian]

Hey @glogiotatidis @axilleas . I wanted to experiment with Ansible as well so I started with this branch and after following @axilleas suggestions I amended some changes here [1] that work locally for me.

@glogiotatidis
I suggest that if you are not planning to continue the work on this branch, we close it for now until we figure out which is the best way to distribute our dev env. Docker [2] also seems promising.

[1] https://github.com/johngian/mozillians/tree/ansible
[2] #1082

[axilleas]

@m8ttyB I don't know much about puppet, but ansible:

• has an easy learning curve
• runs locally, no need for agents
• is python which most mozilla products are
• yaml format is super easy to read

@glogiotatidis @johngian the hard part is to provide a universal distribution method that works at least on Linux/MacOSX. The best viable solution would be Vagrant+Ansible or Vagrant+Docker. More experienced users will be able to bypass vagrant and work only with ansible or docker.