Merge pull request #47 from ARTbio/dev.persistent

Admin user creation, tool installation, data manager and postgresql via supervisor

.gitmodules

@@ -1,15 +1,18 @@
-[submodule "roles/galaxyprojectdotorg.galaxy"]
-	path = roles/galaxyprojectdotorg.galaxy
-	url = git://github.com/galaxyproject/ansible-galaxy
+[submodule "roles/galaxyprojectdotorg.galaxy-tools"]
+	path = roles/galaxyprojectdotorg.galaxy-tools
+	url = git://github.com/galaxyproject/ansible-galaxy-tools.git
 [submodule "roles/galaxyprojectdotorg.galaxy-extras"]
 	path = roles/galaxyprojectdotorg.galaxy-extras
-	url = git://github.com/fabiorjvieira/ansible-galaxy-extras
+	url = git://github.com/galaxyproject/ansible-galaxy-extras
 [submodule "roles/natefoo.postgresql_objects"]
 	path = roles/natefoo.postgresql_objects
 	url = git://github.com/natefoo/ansible-postgresql-objects.git
 [submodule "roles/galaxyprojectdotorg.galaxy-os"]
 	path = roles/galaxyprojectdotorg.galaxy-os
 	url = git://github.com/galaxyproject/ansible-galaxy-os.git
-[submodule "roles/galaxyprojectdotorg.galaxy-tools"]
-	path = roles/galaxyprojectdotorg.galaxy-tools
-	url = git://github.com/galaxyproject/ansible-galaxy-tools
+[submodule "roles/galaxyprojectdotorg.galaxy"]
+	path = roles/galaxyprojectdotorg.galaxy
+	url = git://github.com/galaxyproject/ansible-galaxy.git
+[submodule "ensure_postgresql_up"]
+	path = roles/ensure_postgresql_up
+	url = git://github.com/mvdbeek/ensure_postgresql_up.git

CONTRIBUTORS.md

@@ -3,7 +3,7 @@
 The following individuals have contributed code to ansible-artimed:
 
 * Christophe Antoniewski <drosofff@gmail.com>
-* Fabio Rocha Jimenez Vieira <fabiorjvieira@gmail.com >
+* Fabio Rocha Jimenez Vieira <fabiorjvieira@gmail.com>
 * Marius van den Beek <m.vandenbeek@gmail.com>
 
 # Institutional sponsors

README.md

@@ -1,36 +1,38 @@
 # Requirements
-  * The target Operating System must be a Ubuntu Trusty 64 bits ( it might work on other Debian systems, untested).
+  * The target Operating System must be a Ubuntu Trusty 64 bits (it might work on other Debian systems, untested yet).
   * The target instance must have at least 4GB of RAM.
   * The target user must have sudo rights.
   * You need git and Ansible >= 1.8 (www.ansible.com) on the machine on which you run the playbook.
 
-# Ansible Galaxy instance and NGS tools
+# Ansible Galaxy instance, tools and reference genomes
 To deploy Galaxy, first clone this repository by executing:
 ```
 git clone --recursive https://github.com/ARTbio/ansible-artimed.git
 ```
-Then, if you are targeting a remote machine, adapt the inventory (a sample can be found in the hosts file) and execute:
+Then, if you are targeting a remote machine, adapt the inventory (a sample can be found in the https://github.com/ARTbio/ansible-artimed/blob/master/hosts file) and execute:
 ```
 cd ansible-artimed
-ansible-playbook -i hosts galaxy.yml
+#update/remove "-i hosts" if you want to use or own inventory file
+ansible-playbook -i hosts galaxy.yml 
 ```
-If the ssh key for the target machine is not the default ssh key, add its path to the "ansible_ssh_private_key_file" in the inventory hosts file.
-This procedure will install Galaxy on the target host (refered by the IP on the second line of the file hosts).
-Galaxy will be available on http port 80 (proxy NGINX) on the target host IP.
+If the ssh key for the target is not the default ssh key, add its path to the "ansible_ssh_private_key_file" in the inventory.
+This procedure will install Galaxy on the target.
+Galaxy will be available on http port 80 (proxy NGINX) on the target IP with the admin user as artimed@gmail.com (password: artimed).
+Note that the file https://github.com/ARTbio/ansible-artimed/blob/master/roles/artimed_extras/files/artimed_tool_list.yaml contains the default list of NGS tools to be installed.
+Also note that file https://github.com/ARTbio/ansible-artimed/blob/master/roles/artimed_extras/files/data_managers.yaml contains the default list of reference genomes and data managers to run.
 
-# Installing only Galaxy NGS tools
+# Installing only tools
 This procedure assumes Galaxy has already been installed and configured (for instance with the procedures described above).
-To install only NGS tools on a Galaxy instance, change the value of the variable "install_galaxy" to "False" (located in the file ansible-artimed/hosts) and execute:
-```
-ansible-playbook -i host galaxy.yml
-```
-Note that the file https://github.com/ARTbio/ansible-artimed/blob/master/roles/artimed_extras/files/artimed_tool_list.yaml contains the default list of NGS tools to be installed.
-Therefore, if you want change it, please see the file https://github.com/galaxyproject/ansible-galaxy-tools/blob/master/files/tool_list.yaml.sample for instructions.
-If you want to provide your own list of tools, change the value of the variable "galaxy_tool_list" in ansible-artimed/hosts.
+To install only NGS tools on a Galaxy instance, set the value of the variable "install_galaxy" and "run_data_manager" to "False" in the inventory, and execute the last command of the previous procedure.
+If you want to provide your own list of tools or change the default list, please see the file https://github.com/galaxyproject/ansible-galaxy-tools/blob/master/files/tool_list.yaml.sample for instructions and change the value of the variable "galaxy_tool_list" in the inventory.
+
+# Load only reference genomes
+This procedure assumes Galaxy has already been installed, configured and with data managers installed (for instance with the procedures described above).
+To load reference genomes on a Galaxy instance, change the value of the variables "install_galaxy" and "install_tools" to "False" in the inventory, and execute the last command of the previous procedure.
 
 # Alternative install - Vagrant
 Before continue you must install Vagrant (www.vagrantup.com) and a vagrant compatible Virtual Box (www.virtualbox.org).
-Bring up a vagrant machine using:
+Bring up a vagrant machine by using:
 ```
 vagrant up
 ```
@@ -46,17 +48,19 @@ If you are using Ubuntu on your ansible machine, make sure that you deactivate `
 #Important variables
 The file ansible-artimed/hosts contains various important variables that may be adapated, so you can change it as necessary.
 These variables are:
-- ansible_ssh_user - The login name used to access the target host.
-- ansible_ssh_private_key_file - The ssh private key used to access the target host.
+- ansible_ssh_user - The login name used to access the target.
+- ansible_ssh_private_key_file - The ssh private key used to access the target.
 - install_galaxy - True for install a Galaxy instance.
 - install_tools - True for install the NGS tools.
-- run_dm - True for run the data manager procedure.
+- run_data_manager - True for run the data manager procedure.
 - galaxy_user_name - The Operating System user name for galaxy process.
 - galaxy_server_dir - The home of Operating System user for galaxy process.
 - galaxy_admin - The admin galaxy user.
-- galaxy_api - The api key for tool installation.
+- galaxy_admin_pw - The admin galaxy password.
+- master_api_key - The api key for tool installation and download reference genomes throught galaxy data managers.
 - galaxy_tool_list - The files that constants the list of tools to be installed.
 - galaxy_data_managers - The reference genomes and indexes to be load and build.
 - galaxy_data - The persistent directory where the galaxy config and database directories will be installed or will be recovered.
 - galaxy_database - The persistent directory where postgresql will be installed or will be recovered.
 - galaxy_db - Connection string for galaxy-postgresql.
+- galaxy_changeset_id - The release of Galaxy to be installed (master, dev or release_xx_xx).

galaxy.yml

@@ -10,12 +10,6 @@
     galaxy_config_dir: "{{ galaxy_data }}/config" # Used by artimed_extras and galaxy roles
     galaxy_mutable_data_dir: "{{ galaxy_data }}/database" # Used by artimed_extras and galaxy roles
     galaxy_config_file: "{{ galaxy_config_dir }}/galaxy.ini" # Used by galaxy roles
-
-  pre_tasks:
-    #fix recreation of munge key
-    - shell: rm -rf /tmp/slurm /etc/munge/munge.key
-      when: install_galaxy
-
   roles:
     #installs system package requirements
     - role: galaxyprojectdotorg.galaxy-os
@@ -31,11 +25,12 @@
     #  check_galaxy_data: yes
     #  when: install_galaxy
 
+    - role: ensure_postgresql_up
+
     #this role included in galaxyprojectdotorg.galaxy as a submodule to create galaxy user and its db 
     - role: natefoo.postgresql_objects
       postgresql_objects_users:
         - name: "{{ galaxy_user_name }}"
-          password: "{{ galaxy_user_name }}"
       postgresql_objects_databases:
         - name: "{{ galaxy_user_name }}"
           owner: "{{ galaxy_user_name }}"
@@ -50,43 +45,41 @@
       galaxy_mutable_config_dir: "{{ galaxy_config_dir }}"
       #other vars
       galaxy_manage_database: yes
-      galaxy_fetch_eggs: no
-      galaxy_changeset_id: master
+      # does not actually fetch eggs if galaxy uses pip
+      galaxy_fetch_eggs: yes
       galaxy_vcs: git
+      galaxy_force_checkout: yes
       galaxy_config:
         "app:main":
           admin_users: "{{ galaxy_admin }}"
           database_connection: "{{ galaxy_db }}"
           tool_dependency_dir: "{{ galaxy_server_dir }}/lib/tool_shed/dependencies"
-          master_api_key: "{{ galaxy_master_api_key }}"
+          master_api_key: "{{ master_api_key }}"
           ftp_upload_dir: "{{ galaxy_ftp }}" 
           ftp_upload_site: "ftp {{ galaxy_hostname }}"
           tool_sheds_config_file: "{{ galaxy_config_dir }}/tool_sheds_conf.xml"
           #to be done only in production       
           #debug: False
           #use_interactive: False
+          static_enabled: False
           nginx_x_accel_redirect_base: "/_x_accel_redirect"
           nginx_x_archive_files_base: "/_x_accel_redirect"
           nginx_upload_store: "/tmp"
           nginx_upload_path: "/_upload"
         "uwsgi":
-          stats: "localhost:9191"
-          socket: "localhost:4001"
           master: True
-          logto: "{{ galaxy_data }}/uwsgi.log"
-          pythonpath: lib
-          threads: 4
-          processes: "{{ galaxy_web_processes }}"
       sudo: yes
       sudo_user: "{{ galaxy_user_name }}"
       when: install_galaxy
 
     #installs nginx and proftpd
     - role: galaxyprojectdotorg.galaxy-extras
+      galaxy_uwsgi_static_conf: true
       supervisor_slurm_config_dir: "{{ galaxy_data }}"
       galaxy_root: "{{ galaxy_server_dir }}"
       galaxy_log_dir: "{{ galaxy_data }}"
       galaxy_database_connection: "{{ galaxy_db }}"
+      galaxy_extras_config_scripts: true
       galaxy_extras_install_packages: true
       galaxy_extras_config_nginx: true
       galaxy_extras_config_supervisor: true
@@ -96,7 +89,6 @@
       galaxy_extras_config_slurm: true
       galaxy_extras_config_galaxy_root: true
       galaxy_extras_config_ie_proxy: false
-      supervisor_manage_postgres: false
       supervisor_manage_nginx: true
       supervisor_manage_proftp: true
       supervisor_manage_slurm: true
@@ -107,19 +99,51 @@
       when: install_galaxy
 
     #fix proftpd config file, restarts supervisor, put the test tool shed in the list and create a welcome page
+    - role: ensure_postgresql_up
+
     - role: artimed_extras
       run_extras: true
       pbkdf2: true
+      create_galaxy_admin: true
+      restart_galaxy: true
       when: install_galaxy
 
     #check if tool-data persisted on galaxy (it does nothing if galaxy_data == galaxy_server_dir)
-    #- role: galaxy.movedata
-    #  check_tool_data: yes
-    #  when: install_galaxy
+    - role: galaxy.movedata
+      check_tool_data: yes
+      when: install_galaxy
 
-    - role: galaxyprojectdotorg.galaxy-tools #The default tools are listed in the file files/artimed_tool_list.yaml
-      galaxy_tools_api_key: "{{ galaxy_master_api_key }}"
+    #The default data managers are listed in the file data_manager_list.yaml
+    - role: galaxyprojectdotorg.galaxy-tools 
       galaxy_tools_admin_user: "{{ galaxy_admin }}"
-      galaxy_tools_galaxy_instance_url: "localhost"
+      galaxy_tools_api_key: "{{ master_api_key }}"
+      galaxy_tools_galaxy_instance_url: "{{ galaxy_hostname }}"
+      galaxy_tools_tool_list_files: [ roles/artimed_extras/files/data_manager_list.yaml ]
+      sudo: yes
+      sudo_user: "{{ galaxy_user_name }}"
+      when: run_data_manager
+
+    #Restart Galaxy
+    - role: artimed_extras
+      restart_galaxy: true
+      when: run_data_manager
+
+    #The default tools are listed in the file artimed_tool_list.yaml
+    - role: galaxyprojectdotorg.galaxy-tools 
+      galaxy_tools_admin_user: "{{ galaxy_admin }}"
+      galaxy_tools_api_key: "{{ master_api_key }}"
+      galaxy_tools_galaxy_instance_url: "{{ galaxy_hostname }}"
       galaxy_tools_tool_list_files: [ "{{ galaxy_tool_list }}" ]
+      sudo: yes
+      sudo_user: "{{ galaxy_user_name }}"
       when: install_tools
+
+    #Restart Galaxy
+    - role: artimed_extras
+      restart_galaxy: true
+      when: install_tools
+
+    #Run data managers
+    - role: artimed_extras
+      run_data_managers: true
+      when: run_data_manager

hosts

@@ -1,20 +1,22 @@
 [artimed]
 localhost ansible_ssh_user="root" ansible_ssh_private_key_file="~/.ssh/id_rsa"
-192.54.201.138 ansible_ssh_user="root" ansible_ssh_private_key_file="~/.ssh/id_rsa"
 
 [artimed:vars]
 #install options
 install_galaxy="True"
 install_tools="True"
-run_dm="False"
+run_data_manager="True"
 #params
+supervisor_postgres_database_path="/etc/postgresql/{{ postgresql_version }}/main"
 galaxy_user_name="galaxy"
 galaxy_server_dir="/home/{{ galaxy_user_name }}/{{ galaxy_user_name }}"
 galaxy_admin="artimed@gmail.com"
-galaxy_master_api_key="379ab2a47d714a74b5cee4081703368e"
+galaxy_admin_pw="artimed"
+master_api_key=f137d9af2f2065bd8c5fbadf7111494b
 galaxy_tool_list="roles/artimed_extras/files/artimed_tool_list.yaml"
 galaxy_data_managers="data_managers.yaml"
 #persistent params
-galaxy_data="/home/galaxy/galaxy"
+galaxy_data="/home/{{ galaxy_user_name }}/{{ galaxy_user_name }}"
 galaxy_database="/home/galaxy_database"
 galaxy_db="postgresql:///{{ galaxy_user_name }}?host=/var/run/postgresql"
+galaxy_changeset_id="dev"

roles/artimed_extras/defaults/main.yml

@@ -1,7 +1,12 @@
 run_extras: false
 run_data_managers: false
 pbkdf2: false
+create_galaxy_admin: false
+restart_galaxy: false
+start_galaxy: false
+stop_galaxy: false
 
+admin_user_name: "admin"
 proftpd_conf_path:  "/etc/proftpd/proftpd.conf"
 proftpd_welcome: "Galaxy ARTiMED FTP server!"
 encoding: "base64"

roles/artimed_extras/files/artimed_tool_list.yaml

@@ -1,28 +1,4 @@
 tools:
-- name: data_manager_fetch_genome_all_fasta
-  owner: devteam
-  tool_panel_section_label: 'Data Managers for NGS Mapping'
-- name: data_manager_rsync_g2
-  owner: devteam
-  tool_panel_section_label: 'Data Managers for NGS Mapping'
-- name: data_manager_fetch_genome_dbkeys_all_fasta
-  owner: devteam
-  tool_panel_section_label: 'Data Managers for NGS Mapping'
-- name: data_manager_bwa_mem_index_builder
-  owner: devteam
-  tool_panel_section_label: 'Data Managers for NGS Mapping'
-- name: data_manager_bwa_index_builder
-  owner: devteam
-  tool_panel_section_label: 'Data Managers for NGS Mapping'
-- name: data_manager_bowtie2_index_builder
-  owner: devteam
-  tool_panel_section_label: 'Data Managers for NGS Mapping'
-- name: data_manager_gatk_picard_index_builder
-  owner: devteam
-  tool_panel_section_label: 'Data Managers for NGS Mapping'
-- name: data_manager_sam_fasta_index_builder
-  owner: devteam
-  tool_panel_section_label: 'Data Managers for NGS Mapping'
 - name: featurecounter
   owner: devteam
   tool_panel_section_label: 'NGS Mapping'

roles/artimed_extras/files/data_manager_list.yaml

@@ -0,0 +1,49 @@
+tools:
+- name: data_manager_picard_index_builder
+  owner: devteam
+  tool_panel_section_label: 'Data Managers for NGS Mapping'
+- name: data_manager_build_kraken_database
+  owner: devteam
+  tool_panel_section_label: 'Data Managers for NGS Mapping'
+- name: data_manager_fetch_ncbi_taxonomy
+  owner: devteam
+  tool_panel_section_label: 'Data Managers for NGS Mapping'
+- name: data_manager_gemini_database_downloader
+  owner: iuc
+  tool_panel_section_label: 'Data Managers for NGS Mapping'
+- name: data_manager_hisat2_index_builder
+  owner: iuc
+  tool_panel_section_label: 'Data Managers for NGS Mapping'
+- name: data_manager_snpeff
+  owner: iuc
+  tool_panel_section_label: 'Data Managers for NGS Mapping'
+- name: data_manager_diamond_database_builder
+  owner: bgruening
+  tool_panel_section_label: 'Data Managers for NGS Mapping'
+- name: region_motif_data_manager
+  owner: jeremyjliu
+  tool_panel_section_label: 'Data Managers for NGS Mapping'
+- name: data_manager_fetch_genome_all_fasta
+  owner: devteam
+  tool_panel_section_label: 'Data Managers for NGS Mapping'
+- name: data_manager_rsync_g2
+  owner: devteam
+  tool_panel_section_label: 'Data Managers for NGS Mapping'
+- name: data_manager_fetch_genome_dbkeys_all_fasta
+  owner: devteam
+  tool_panel_section_label: 'Data Managers for NGS Mapping'
+- name: data_manager_bwa_mem_index_builder
+  owner: devteam
+  tool_panel_section_label: 'Data Managers for NGS Mapping'
+- name: data_manager_bwa_index_builder
+  owner: devteam
+  tool_panel_section_label: 'Data Managers for NGS Mapping'
+- name: data_manager_bowtie2_index_builder
+  owner: devteam
+  tool_panel_section_label: 'Data Managers for NGS Mapping'
+- name: data_manager_gatk_picard_index_builder
+  owner: devteam
+  tool_panel_section_label: 'Data Managers for NGS Mapping'
+- name: data_manager_sam_fasta_index_builder
+  owner: devteam
+  tool_panel_section_label: 'Data Managers for NGS Mapping'

roles/artimed_extras/files/data_managers.yaml

@@ -0,0 +1,11 @@
+dbkeys:
+- dbkey: "hg19"
+- dbkey: "hg38"
+data_managers:
+- id: "toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/data_manager_fetch_genome_all_fasta/0.0.1"
+- id: "toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/data_manager_fetch_genome_dbkeys_all_fasta/0.0.1"
+- id: "toolshed.g2.bx.psu.edu/repos/devteam/data_manager_bowtie2_index_builder/data_manager_bowtie2_index_builder/0.0.1"
+- id: "toolshed.g2.bx.psu.edu/repos/devteam/data_manager_bwa_index_builder/data_manager_bwa_index_builder/0.0.1"
+- id: "toolshed.g2.bx.psu.edu/repos/devteam/data_manager_bwa_mem_index_builder/data_manager_bwa_mem_index_builder/0.0.1"
+- id: "toolshed.g2.bx.psu.edu/repos/devteam/data_manager_gatk_picard_index_builder/data_manager_gatk_picard_index_builder/0.0.1"
+- id: "toolshed.g2.bx.psu.edu/repos/devteam/data_manager_sam_fasta_index_builder/data_manager_sam_fasta_index_builder/0.0.1"