ome-dundeeomero next steps / cleanup #273

kennethgillen · 2017-05-19T09:20:57Z

Next steps and cleanup for the New Nightshade server pair.

kennethgillen · 2017-05-22T10:32:44Z

Full playbook run of 'nightshade-web' tested on a test system (infra-testpr), installs from pypi omero-figure and FPBioimage viewer.

kennethgillen · 2017-05-22T14:51:14Z

NFS remote processing mount added to playbook and deployed on live system. Tested working.

kennethgillen · 2017-06-06T15:26:23Z

@joshmoore - note the addition of an upgrade_webapps == "True" - cli-flag based upgrade for safety. Else 'state: present'.

kennethgillen · 2017-06-07T16:01:59Z

If any @openmicroscopy/devops would like to have a look-over that'd be good.

I merged to document the changes made to the live system, to continue work on the ome-demoserver, but it's still worth an 👀 from others.

manics · 2017-06-08T09:13:55Z

ansible/server-state-playbooks/nightshade-web/playbook.yml

-        - reload systemd
+    # Set desired state of webapps (upgrade v.s. install/keep current version)
+    - set_fact:
+        webapps_state: present 


FYI set_fact will set a host-var that may persist across playbooks in the same ansible-playbook command, so there's a (small) danger of inadvertent variable conflicts, especially with shorter names and more complex deployments. Since this var is only used in one place you could get rid of it completely and in name: Omero.web plugins | plugin install via pip & pypi have state: "{{ webapps_state | default('present') }}", and pass -e webapps_state=latest on the command line to upgrade.

Yep - thanks, I couldn't find how to do make that conditional just work, without registering the fact. I read that indeed it would persist, so made sure to initialise it first with no conditional to present. Thanks for the syntax, and I'll refactor this as you suggest.

manics · 2017-06-08T09:17:08Z

ansible/server-state-playbooks/nightshade-web/playbook.yml

      no_log: true

+    # This to be refactored for RHEL/non-spacewalk at some point


If you refactor I can think of two options:

Install/configure openmicroscopy.system-monitor-agent in each server's playbook as at present

Remove all mentions of openmicroscopy.system-monitor-agent in individual playbooks and have a single site-wide playbook that handles internal monitoring on all servers.

I've found a better way which will work with RHEL/non-spacewalk servers - by checking for the SLS nameservers in the networking config - this was introduced on the ome-dundeeomero work and continues. These are good ideas for further refactoring though, so thank you.

manics · 2017-06-08T09:23:05Z

ansible/server-state-playbooks/ome-dundeeomero/templates/nightly-pg_dump-omero.sh.j2

+# Clean the files from the folder-format dump directory 
+rm {{ omero_server_db_dumpdir_parent }}/{{ omero_server_db_dumpdir_name }}/* 
+# Create a fresh db dump
+sudo -u postgres pg_dump -j {{ (ansible_processor_count * ansible_processor_cores) |round|int }} -Fd -f {{ omero_server_db_dumpdir_parent }}/{{ omero_server_db_dumpdir_name }} {{ vault_omero_server_dbname }}


Why do you need |round|int?

Jinja2 was creating doubles - or certainly values with a decimal point, which at least nginx did not appreciate.

Is that because you were divding by 2? https://github.com/openmicroscopy/infrastructure/blob/bb9eff03b7e18452424266a32e0d1691a94d0b23/ansible/server-state-playbooks/nightshade-web/playbook.yml#L71

manics · 2017-06-08T09:24:47Z

ansible/server-state-playbooks/ome-dundeeomero/playbook.yml

+    - name: PostgreSQL Nightly Backups | Create the backups directory
+      become: yes
+      file:
+        path: "{{ omero_server_db_dumpdir_parent }}/{{ omero_server_db_dumpdir_name }}"


Minor point, the omero_server_ prefix is used by the omero-server role, so there's potential for confusion between vars specific to this playbook and those in the role.

manics · 2017-06-08T09:25:40Z

ansible/server-state-playbooks/ome-dundeeomero/playbook.yml

@@ -11,6 +11,27 @@
           ((ansible_virtualization_type is defined)
           and (ansible_virtualization_type == "VMware"))

+    # Not required for an OMERO server install - UoD specific implementation


Does https://galaxy.ansible.com/openmicroscopy/nfs-mount/ work for this mount?

Not sure, didn't see/try it. What is there just now certainly worked.

Fair enough, but that's certainly the point of the reviews is so that we express to the next person coming along here's how we think it should be done.

Agreed - I definitely do see the value of these reviews - I didn't skip review because I thought otherwise - rather to progress ome-demoserver and have all my ansible code in one (new) branch such that I could diff / merge from my previous code.

manics · 2017-06-08T09:28:25Z

ansible/server-state-playbooks/nightshade-web/playbook.yml

@@ -1,35 +1,16 @@
 # Install OMERO.web with a public user on localhost
+# Pass the command line 'upgrade-webapps' flag to upgrade webapps


See my later comment, but if you want to keep it as it is there's a typo: _ not -

manics · 2017-06-08T09:30:37Z

ansible/server-state-playbooks/nightshade-web/templates/omero-web-config-for-webapps.j2

@@ -2,7 +2,23 @@
 # Add web-extension OMERO.web configuration
 # after web already installed and running via role

-config set -- omero.web.apps '["figure"]'
-config set -- omero.web.ui.top_links '[["Data", "webindex", {"title": "Browse Data via Projects, Tags etc"}], ["History", "history", {"title": "History"}], ["Help", "http://help.openmicroscopy.org/", {"target": "new", "title": "Open OMERO user guide in a new tab"}], ["Figure", "figure_index", {"target": "new", "title": "Open OMERO.Figure in a new tab"}] ]'


Minor point: there's no templating in this file (apart from ansible_managed) so it could be a normal file and copy.

My 2c: I don't mind all files being templates.

It certainly helps when merging >1 playbook with some plain copies and some templating. A 'file' is a template which just doesn't change very much?

Yes, unless you have a string that contains {{ or {%

Good tip, thanks.

manics · 2017-06-08T09:32:24Z

ansible/server-state-playbooks/nightshade-web/playbook.yml

@@ -42,18 +23,35 @@
      lvm_lvfilesystem: "{{ provision_root_filesystem }}"

    # OMERO.web configuration in host_vars in different repository


So omero_web_config_set is in hostvars, but omero_server_config_set is in ome-dundeeomero/playbook.yml?

As I became more confident with the Ansible, I realised I could define the vars in the playbook, and keep as much as possible public, only pulling the individual private variables from private config. Initially I thought I'd have to keep the whole omero_web_config_set together as one. ome-dundeeomero represents my having learned that lesson. A refactor of the ns-web playbook would be nice, in this style.

Removing ugrade flags.

e500d83

kennethgillen added the ON_HOLD label May 19, 2017

kennethgillen added 3 commits May 19, 2017 21:39

Creating DB dump folder and script to run the daily dumps

3529dd9

omero-fpbioimage - plugin install

dc33e37

Switch from manual install of figure to pypi install

d5a195e

kennethgillen added ON_HOLD and removed ON_HOLD labels May 22, 2017

kennethgillen added 2 commits May 22, 2017 13:56

The required webapp config

2a5b0ff

NGOM mount for remote processing

dc0b57e

name update

8a4e685

kennethgillen removed the ON_HOLD label May 22, 2017

Adding iviewer & tag apps

1164b3c

kennethgillen merged commit bb9eff0 into ome:master Jun 6, 2017

kennethgillen deleted the dundeeomero-next_steps branch June 6, 2017 15:32

kennethgillen requested review from joshmoore and manics June 7, 2017 14:13

manics reviewed Jun 8, 2017

View reviewed changes

joshmoore mentioned this pull request Dec 1, 2017

Review infra PRs for modifications ome/prod-playbooks#16

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ome-dundeeomero next steps / cleanup #273

ome-dundeeomero next steps / cleanup #273

kennethgillen commented May 19, 2017

kennethgillen commented May 22, 2017

kennethgillen commented May 22, 2017

kennethgillen commented Jun 6, 2017

kennethgillen commented Jun 7, 2017

manics Jun 8, 2017

kennethgillen Jun 8, 2017

manics Jun 8, 2017

kennethgillen Jun 8, 2017

manics Jun 8, 2017

kennethgillen Jun 8, 2017

manics Jun 9, 2017

manics Jun 8, 2017

manics Jun 8, 2017

kennethgillen Jun 8, 2017

joshmoore Jun 8, 2017

kennethgillen Jun 8, 2017

manics Jun 8, 2017

manics Jun 8, 2017

joshmoore Jun 8, 2017

kennethgillen Jun 8, 2017

manics Jun 9, 2017

kennethgillen Jun 9, 2017

manics Jun 8, 2017

kennethgillen Jun 8, 2017

		no_log: true

		# This to be refactored for RHEL/non-spacewalk at some point

		@@ -1,35 +1,16 @@
		# Install OMERO.web with a public user on localhost
		# Pass the command line 'upgrade-webapps' flag to upgrade webapps

		@@ -42,18 +23,35 @@
		lvm_lvfilesystem: "{{ provision_root_filesystem }}"

		# OMERO.web configuration in host_vars in different repository

ome-dundeeomero next steps / cleanup #273

ome-dundeeomero next steps / cleanup #273

Conversation

kennethgillen commented May 19, 2017

kennethgillen commented May 22, 2017

kennethgillen commented May 22, 2017

kennethgillen commented Jun 6, 2017

kennethgillen commented Jun 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment