Skip to content

jonathanmorgan/ansible-patterns

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

128 Commits
host_vars
host_vars
 
 
roles
roles
 
 
.gitignore
.gitignore
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
hosts.yml
hosts.yml
 
 
only_apache.yml
only_apache.yml
 
 
only_common.yml
only_common.yml
 
 
only_context_dev.yml
only_context_dev.yml
 
 
only_django_project.yml
only_django_project.yml
 
 
only_jupyterhub.yml
only_jupyterhub.yml
 
 
only_mariadb.yml
only_mariadb.yml
 
 
only_new_server.yml
only_new_server.yml
 
 
only_postgresql.yml
only_postgresql.yml
 
 
only_r.yml
only_r.yml
 
 
only_rstudio.yml
only_rstudio.yml
 
 
only_sourcenet_dev.yml
only_sourcenet_dev.yml
 
 
research.yml
research.yml
 
 
server_apache.yml
server_apache.yml
 
 
server_common.yml
server_common.yml
 
 
server_jupyterhub.yml
server_jupyterhub.yml
 
 
server_postgresql.yml
server_postgresql.yml
 
 
server_rstudio.yml
server_rstudio.yml
 
 

Repository files navigation

Ansible script patterns

DOI

Setup

Ansible - Installation on control machine

  • on mac or linux, do pip install ansible.

  • For any OS with anaconda Python distribution, I make a separate conda environment named ansible, since it requires pip to install, so you keep it separate from other packages installed with conda (mixing conda and pip generally means trouble over time).

    • if on catalina, might need to open bash from zsh shell, and if so, will likely also need to move your anaconda init from ~/.bash_profile to ~/.bashrc.
    • if needed, activate anaconda: source ~/miniconda3/bin/activate
    • conda create --name ansible python=3
    • conda activate ansible
    • pip install ansible

  • for other ways to install: http://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html

Ansible - Installation on a target

  • http://docs.ansible.com/ansible/latest/user_guide/intro_getting_started.html

  • put your control machine’s SSH public key in ~/.ssh/authorized_keys on the target machine in the home folder of the user you'll use to run ansible commands on the target. (permissions: ~/.ssh folder should be 700; authorized_keys should be 600).

  • make sure that python 2 or 3 are installed. From installation guide:

    • By default, Ansible uses the python interpreter located at /usr/bin/python to run its modules. However, some Linux distributions may only have a Python 3 interpreter installed to /usr/bin/python3 by default. On those systems, you may see an error like:

      • "module_stdout": "/bin/sh: /usr/bin/python: No such file or directory\r\n"

    • you can either set the ansible_python_interpreter inventory variable (see Working with Inventory) to point at your interpreter or you can install a Python 2 interpreter for modules to use. You will still need to set ansible_python_interpreter if the Python 2 interpreter is not installed to /usr/bin/python.

  • if it is a new installation, also update /etc/hosts so that it includes your host name in the entry for “127.0.0.1” (example of having added hostname “research”):

    • 127.0.0.1 localhost.localdomain localhost research
    • NOTE: if your hostname (run hostname to see what it is) isn’t a for-real DNS name, if you don’t update /etc/hosts to include your server name (example: “research”) in your set of DNS names for 127.0.0.1, sudo will be SLOW: https://ubuntuforums.org/showthread.php?t=1155261

  • And, a random note: If you are installing on a local VM where you'll assign an IP address (I ran into it in VMWare Fusion), but have no DNS name, but you want to make a DNS name by adding it to /etc/hosts on your host computer, don't put underscores in the host name you make up. Far as I can tell, Apache 2.4 will not accept requests to the hostname with underscores in it, but will accept requests to the IP address of the VM, and will accept requests to hostnames with no underscores. There might be a way to get this to work, but I do not know it. So, for example:

    • GOOD: ubuntu.local
    • BAD: ubuntu_server_lts.local

research patterns

Research server playbook is research.yml.

To fire up a sourcenet/context server, do the research.yml playbook, then the only_context_dev.yml playbook.

research quick start

The following steps will install all services on the server whose DNS name is research.local:

  • go into ansible-patterns/host_vars and rename research.local.yml to research.local. Inside, make sure to set:

    • ansible_user
    • ansible_become_password
    • server_ip_address
    • server_webroot_folder (defaults to ubuntu default of /var/www/html)
    • server_cpu_type ("amd64" for intel, "arm64" for M1 macs)
    • jupyterhub_configproxy_auth_token (32 character random hexadecimal number: openssl rand -hex 32)

  • set research.local to point to the IP address of the server you want to install to in your hosts file (/etc/hosts on unix-like machines).

  • run the "research.yml" playbook: in ansible-patterns root, run ansible-playbook research.yml -i ./hosts.yml

    • for the cautious, here is a list of the roles in research.yml, in order, along with example command to run each separately:

      • new_server (ansible-playbook only_new_server.yml -i ./hosts.yml)

      • common (ansible-playbook only_common.yml -i ./hosts.yml)

      • apache (ansible-playbook only_apache.yml -i ./hosts.yml)

      • postgresql (ansible-playbook only_postgresql.yml -i ./hosts.yml)

      • jupyterhub (ansible-playbook only_jupyterhub.yml -i ./hosts.yml)

      • If not arm64 (so if not M* mac):

        • r (ansible-playbook only_r.yml -i ./hosts.yml)
        • RStudio Server (ansible-playbook only_rstudio.yml -i ./hosts.yml)

      • django_project (ansible-playbook only_django_project.yml -i ./hosts.yml)

  • To also install the context set of applications, run the research.yml playbook, then the only_sourcenet_dev.yml playbook: ansible-playbook only_sourcenet_dev.yml -i ./hosts.yml

  • If you just install everything for the django application inside the ansible user's home folder (the default), you'll then need to change permissions so Apache can do what it needs to do to serve the django application:

      cd /home
  chmod 755 <ansible_user>
  cd <ansible_user>
  chmod 755 work
  cd work
  chmod 755 django
  cd django
  chmod 775 <django_project_name>
  cd <django_project_name>
  find . -type d -exec chmod 775 {} \;
    • I altered the lines above with 775 to that, from 777, to see if that breaks things. If so, re-run the two commands above that contain "775", but replace "775" with "777".
    • NOTE: DO NOT DO THIS IN A PRODUCTION SYSTEM. Instead, place the django project folder in a shared place you can give the "apache" user and any developers access to.
    • Also, I tried to put "apache" user into the ansible user's group, then just set permissive group permissions, but that didn't work for some reason...

This quick start assumes that you are making a server whose DNS name is "research.local". If you want a different DNS name, for example "ubuntu.local":

  • when you copy research.local.yml, name the copy the DNS name you want (ubuntu.local).

  • Inside your copy of research.local.yml (named ubuntu.local for this example):

    • set server_host_name to your server's internal host name - "ubuntu"?
    • set server_name to your server's DNS name for external connections - "ubuntu.local" in this example.

  • Update the file hosts.yml to substitute your preferred DNS name ("ubuntu.local" in this example) for all mentions of "research.local".

research notes

  • must update:

    • hosts.yml

      • need to update to match the particular layout of servers you intend to use.
      • default is all installed on the same server, named research.local.
      • Note: more complex configurations will require manual configuration beyond what these scripts provide.

    • /host_vars

      • each host needs a file named its DNS name to hold variables.

      • use the research.local.yml file as a template for variables related to each of your hosts.

      • This includes:

        • ansible_user
        • ansible_become_password
        • server_ip_address
        • jupyterhub_configproxy_auth_token (32 character random hexadecimal number: openssl rand -hex 32)

      • Notes on specific variables:

        • django_superuser_password: The command that sets the django superuser password uses a shell script, and so the password value here can't contain:
          • "$"
          • open square bracket without closed square bracket ("[").

    • vault password(s)

      • if you want to encrypt either files or variable values within yml files, you'll need to have one or more vault passwords to use as the symmetric encryption key.

      • if you need your deployment to run unattended, you'll want to create one or more files for your vault passwords, then direct the ansible-playbook command to them when you deploy.

        • these can be located wherever you want.

    • ansible config:

      • one way to store a single vault password is to store its location in ansible config, in vault_password_file.

      • default locations:

        • /etc/ansible/ansible.cfg
        • ~/.ansible.cfg

    • if installing rstudio, roles/rstudio/tasks/rstudio_variables.yml

      • add rstudio version to download (will have latest as of last changes to that file, might be outdated).

Ansible Notes

Getting started

inventory file

  • Inventory file is where ansible looks to see which servers it is controlling.

  • default location is /etc/ansible/hosts

    • on work computer, mine is in /Users/jonathanmorgan/Documents/work/ansible/research

  • when you run ansible, you can specify a different file with the -i <inventory_file_path> command.

  • ansible inventory and playbook files are generally implemented in YAML. Example simple inventory:

      all:
    vars:
      ansible_user: jonscomo
      ansible_python_interpreter: /usr/bin/python3
    hosts:
      172.16.111.133
  all:
    hosts:
      172.16.111.133:
        ansible_user: jonscomo
        ansible_python_interpreter: /usr/bin/python3

  • any grouping from “all” down can have a set of “vars” associated with it.

  • if a variable is defined in multiple places, you can also

  • specific variables:

    • ansible_user: user ansible uses when connecting.

    • ansible_python_interpreter: path to Python to use (either in general, if in all—>vars, or per host if under host name).

    • you can also specify variable values for a given host in a separate file:

      • host variable files are located in ./host_vars, relative to the location of your playbook.

      • each host’s variables are stored in a file named just their host name inside host_vars.

      • variables are stored in the root of a YAML file. Simple example:

          # default variables for host research.local
  ansible_user: jonathanmorgan
  ansible_python_interpreter: /usr/bin/python3
  ansible_become: yes
  ansible_become_user: root
  ansible_become_password: today123

  • details: http://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html

ansible command basics

  • once you have inventory file, the most basic tests are to ping, then run a simple command:

      ansible all -m ping -i ./hosts.yaml
  ansible all -a "/bin/echo hello" -i ./hosts.yaml

  • options:

    • -i <inventory_file_path>
    • -u <username> (on remote machine)

  • sudo examples:

      # as bruce
  $ ansible all -m ping -u bruce
  # as bruce, sudoing to root
  $ ansible all -m ping -u bruce --sudo
  # as bruce, sudoing to batman
  $ ansible all -m ping -u bruce --sudo --sudo-user batman
   
  # With latest version of ansible `sudo` is deprecated so use become
  # as bruce, sudoing to root
  $ ansible all -m ping -u bruce -b
  # as bruce, sudoing to batman
  $ ansible all -m ping -u bruce -b --become-user batman

overview

configuration

playbooks

Roles

  • Documentation: https://docs.ansible.com/ansible/latest/user_guide/playbooks_reuse_roles.html#roles

  • Roles are sets of tasks that can be combined in different ways to create various configurations on a particular server or set of servers.

  • Roles by default are stored in a folder named “roles” in the directory where your playbook lives. Each role’s folder includes:

    • /handlers - handler definitions (reusable tasks that can be invoked from tasks lists) for the role. Main file is “main.yml”, then it can include others if you want to separate handlers out for organization.
    • /tasks - list of tasks to implement for a given role. The main file is “main.yml”, then it can include others if you want to separate handlers out for organization.
    • /templates - files that are templates for configuration or other uses for the role.
    • /vars - files to hold variables for a role. Main file is “main.yml”.

  • A playbook decides which roles are invoked based on role tasks. A given role task will contain a name (of course), a “hosts” field that either includes a host, a list of hosts, or a hosts group from your inventory file, and then the role to run against all matching hosts.

    • Example inventory file:

        all:
    hosts:
      research.local
    children:
      new_servers:
        hosts:
          research.local

    • Example role task to run the “new_server” role against servers in “new_servers” group:

        - name: set up new servers
    hosts: new_servers
    roles:
      - new_server
      • This combination will cause the tasks in ./roles/new_server/tasks/main.yml to be run against the hosts in group “new_servers”, so in this example, just the host “research.local”.

Vault

  • Allows you to encrypt either variable values or files using a passphrase.

  • General vault documentation: http://docs.ansible.com/ansible/latest/user_guide/vault.html

  • ansible-vault command (https://docs.ansible.com/ansible/latest/cli/ansible-vault.html) lets you encrypt files or variable values within a value based on a secret encryption password.

  • to start, you need a vault password. This can be provided on the command line, or stored in a file.

    • create a strong password.

    • store the password and only the password in the first and only line of a file in your user's home folder named ".ansible_vault" (so "~/.ansible_vault").

    • all share same password:

      • command line (all share same password):

        • ansible-playbook site.yml --vault-id ~/.ansible_vault

      • in config INI-format file (either /etc/ansible/ansible.cfg or ~/.ansible.cfg by default, or provided by the -c option), create an ansible configuration file that contains the following:

          [defaults]
  vault_password_file = <absolute_path_to_vault_password_file>

    • different password per ID (command line only):

      • ansible-playbook site.yml --vault-id dev@~/.ansible_vault --vault-id prod@~/.ansible_prod_vault

  • basics:

    • https://docs.ansible.com/ansible/latest/user_guide/playbooks_vault.html
    • encrypt a value to include in a YAML file variable:

      • http://docs.ansible.com/ansible/latest/cli/ansible-vault.html#ansible-vault-encrypt-string

      • ansible-vault encrypt_string --vault-password-file <path_to_password_file> --encrypt-vault-id default '<string_to_encrypt>' --name '<variable_name>'

      • Example: ansible-vault encrypt_string --vault-password-file ~/.ansible_vault --encrypt-vault-id default '<password>' --name ansible_become_password

      • Example output:

          ansible_become_password: !vault |
  $ANSIBLE_VAULT;1.1;AES256
  35633331303937613165656461336336343638643562383435383162303662663634366337353730
  3535666630666130656238313630363662383766303131350a303031613934646339313836393932
  30353838646265663535653036353339633664383033623338316236353439613930396335653634
  6332653732643937330a336532363265313731636538646438396162656666366439303561316465
  6265 
  Encryption successful

  • multiple vault passwords:

Variables

Tasks:

Task Modules

apache2_module module

apt module

  • Documentation: https://docs.ansible.com/ansible/latest/modules/apt_module.html

    • Install one:

        - name: install dams
    become: yes
    become_user: root
    apt:
      name: dkms
      state: present

    • Install multiple:

        - name: install OpenSSH
    become: yes
    become_user: root
    apt:
  name:
    - openssh-client
    - openssh-server
    - openssh-sftp-server
  state: present

    • Update cache:

        # update cache
  - name: Update apt cache
    become: yes
    become_user: root
    apt:
      update_cache: yes

    • Dist upgrade:

        # upgrade
  - name: Upgrade installed packages
    become: yes
    become_user: root
    apt:
      upgrade: dist

apt_key module

copy module

django_manage module

  • Documentation:

  • Examples:

    • idempotent (only if doesn't already exist) superuser create:

      • https://gist.github.com/elleryq/9c70e08b1b2cecc636d6

      • and, see the django_project role in https://github.com/jonathanmorgan/ansible-patterns.

          # check if superuser user exists, if not, create user.
  - name: Check if superuser user exists, if not, create user.
    django_manage:
      command: shell -c "from django.contrib.auth import get_user_model; MyUser = get_user_model(); MyUser.objects.filter( username__exact = '{{ django_superuser_username }}' ).count() == 0 or exit(); new_super_user = MyUser( username = '{{ django_superuser_username }}', password='{{ django_superuser_password }}', email='{{ django_superuser_email }}', is_superuser = True, is_staff = True ); new_super_user.save();"
      app_path: "{{ django_project_folder_path }}"
      virtualenv: "{{ django_virtualenv_path }}"

    • migrate without django_manage:

        # run `python manage.py migrate`
  - name: run `python manage.py migrate`
    shell: "{{ django_virtualenv_bin_folder }}/python manage.py migrate"
    args:
      chdir: "{{ django_project_folder }}"

    • migrate with django_manage:

        - name: run `python manage.py migrate`
    django_manage:
      app_path: "{{ django_project_folder_path }}"
      command: migrate
      virtualenv: "{{ django_virtualenv_path }}"

file module

get_url module

git module

  • Used to interact with git repositories.

  • Documentation: https://docs.ansible.com/ansible/latest/modules/git_module.html

  • Examples:

    • clone a repository:

        # prerequisite: python_utilities
  # - https://github.com/jonathanmorgan/python_utilities.git
  # - git@github.com:jonathanmorgan/python_utilities.git
  - name: clone python_utilities
    git:
      repo: https://github.com/jonathanmorgan/python_utilities.git
      dest: "{{ django_project_folder_path }}/python_utilities"

lineinfile module

mysql_db module

mysql_user module

  • Documentation: https://docs.ansible.com/ansible/latest/modules/mysql_user_module.html#mysql-user-module

  • Example:

    • delete all anonymous users (all hosts):

        # Delete anonymous users
  - name: Deletes anonymous server user
    mysql_user:
      login_user: root
      login_password: "{{ server_mariadb_root_password }}"
      name: ""
      host_all: yes
      state: absent
    notify: restart mysql
    tags: mariadb

    • create new non-root admin user:

        # create non-root admin user?
  # Creates database user {{ server_mariadb_admin_user }} and password {{ server_mariadb_admin_password }} with all database privileges and 'WITH GRANT OPTION'
  - name: Create non-root admin user {{ server_mariadb_admin_user }}
    mysql_user:
      login_user: root
      login_password: "{{ server_mariadb_root_password }}"
      name: "{{ server_mariadb_admin_user }}"
      password: "{{ server_mariadb_admin_password }}"
      host: "{{ item }}"
      priv: '*.*:ALL,GRANT'
      state: present
    with_items:
      - "%"
      - ::1
      - 127.0.0.1
      - localhost
    when:
      - server_mariadb_admin_user != ""
      - server_mariadb_admin_password != ""
    notify: restart mysql
    tags: mariadb

npm module

openssl_* modules

pip module

  • Documentation: http://docs.ansible.com/ansible/latest/modules/pip_module.html
  • Examples:
  • can run a particular pip executable

    • under pip, include executable: <path_to_pip>. This could be just pip3, or could be something like /usr/bin/pip3.

    • Example:

        # ==> virtualenv and virtualenvwrapper
  - name: install virtualenv and virtualenvwrapper
    become: yes
    become_user: root
    pip:
      executable: pip3
      name:
        - virtualenv
        - virtualenvwrapper
  • supports virtualenvs

    • under pip, include the “virtualenv: <path_to_virtualenv_folder>”.

    • If virtualenv not initialized, will first initialize at path you specify.

    • Example:

        # Install specified python requirements in indicated (virtualenv).
  - pip:
      requirements: /my_app/requirements.txt
      virtualenv: /my_app/venv

postgresql_db module

  • Documentation: https://docs.ansible.com/ansible/latest/modules/postgresql_db_module.html

  • Examples:

    • create with login username and password (to get around oddness with using postgres user):

        # create database for django app in postgres
  - name: create postgresql database for django_project named {{ django_db_name }}
    postgresql_db
      login_user: "{{ server_pgsql_admin_user }}"
      login_password: "{{ server_pgsql_admin_password }}"
      db: postgres
      name: "{{ django_db_name }}"
      owner: "{{ django_db_username }}"

postgresql_user module

  • Documentation: https://docs.ansible.com/ansible/latest/modules/postgresql_user_module.html

  • Notes:

    • requires the psycopg2 module be installed in the python environment that ansible is using to do install tasks.
    • requires encrypted passwords in PostgreSQL 10 - not sure of implications of this.

  • Errors:

    • on ubuntu, if you get error: FAILED! => {"changed": false, "msg": "unable to connect to database: FATAL: Peer authentication failed for user "postgres"\n"}

      • try to install package “acl” on ubuntu and try again. Didn’t work for me.

      • for this (no clue why):

          - name: Create non-postgres admin user
    become: yes
    become_user: postgres
    postgresql_user:
      name: '{{server_pgsql_admin_user}}'
      password: '{{server_pgsql_admin_password}}'
      role_attr_flags: SUPERUSER
      encrypted: yes
    when:
      - server_pgsql_admin_user != ""
      - server_pgsql_admin_password != ""
              * Alternative (not pretty, but it works):

  # create non-postgres admin user?
  - name: Create non-postgres admin user
    become: yes
    become_user: root
    shell: sudo -u postgres psql -c "DO \$do\$ BEGIN IF NOT EXISTS( SELECT * FROM pg_catalog.pg_roles WHERE rolname = '{{server_pgsql_admin_user}}') THEN CREATE USER {{server_pgsql_admin_user}} WITH PASSWORD '{{server_pgsql_admin_password}}' SUPERUSER; END IF; END \$do\$;"
    when:
      - server_pgsql_admin_user != ""
      - server_pgsql_admin_password != “"

service module

  • Documentation: https://docs.ansible.com/ansible/latest/modules/service_module.html

  • Notes:

  • Examples:

    • Enable new systems service:

        # Create systemd service for jupyterhub.
  - name: Create systemd service for jupyterhub
    become: yes
    become_user: root
    template:
      src: jupyterhub.service.j2
      dest: /etc/systemd/system/jupyterhub.service
  # Enable jupyterhub service - have to say use “service”, systemd service has bug.
  # https://github.com/ansible/ansible-modules-core/issues/3764
  - name: Enable jupyterhub service
    become: yes
    become_user: root
    service:
      name: jupyterhub
      enabled: True
      use: service
  # Start jupyterhub and set as a startup service
  - name: Enable jupyterhub as a startup service
    become: yes
    become_user: root
    service:
      name: jupyterhub
      state: started
      enabled: yes
          * Restart Apache:
  # restart apache
  - name: restart apache2
    become: yes
    become_user: root
    service:
      name: apache2
      state: restarted

shell module

stat module

template module

ufw module

uri module

ansible modules:

TODO

todo:

  • jupyter kernel-ize research virtualenv.
  • git config
  • set up one or more standard users, including git config, Downloads folder, etc.

About

Ansible scripts for services and servers (postgresql, R, rstudio, jupyterhub, django, etc.)

Resources

Readme

License

LGPL-3.0 license
Activity

Stars

1 star

Watchers

1 watching

Forks

0 forks
Report repository

Releases 3

ansible-patterns - ansible scripts for installing research tools Latest
Oct 30, 2019
+ 2 releases

Packages

No packages published

Contributors 2

  •  
  •  

Languages