[xenial] Add documentation for Xenial upgrade #4140

eloquence · 2019-02-15T01:36:40Z

This PR adds instructions for

performing an upgrade to Ubuntu 16.04 in-place
re-installing from a backup (on existing or new hardware)

(The bulk of the work here was done by @zenmonkeykstop with copyediting by yours truly.)

Resolves #4057.

How to test

A tester should follow the instructions as closely as possible, against supported hardware.

Status

Ready for review

make docs-lint passes

codecov-io · 2019-02-20T03:18:51Z

Codecov Report

Merging #4140 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #4140   +/-   ##
========================================
  Coverage    84.79%   84.79%           
========================================
  Files           43       43           
  Lines         2782     2782           
  Branches       303      303           
========================================
  Hits          2359     2359           
  Misses         355      355           
  Partials        68       68

Impacted Files	Coverage Δ
securedrop/securedrop/i18n_tool.py	`94.14% <0%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 66452c3...6d70fb3. Read the comment docs.

heartsucker

Couple of minor things, otherwise lgtm.

heartsucker · 2019-02-22T08:59:49Z

docs/upgrade/xenial_backup_install_restore.rst

+the servers during the OS installation.
+
+If you are planning to install the Xenial instance on new hardware, make sure
+that you have all the necessary hardware to hand and configured. This guide


to hand -> on hand

heartsucker · 2019-02-22T09:06:51Z

docs/upgrade/xenial_upgrade_in_place.rst

+- on the order of 8 hours if no issues are encountered, and longer if anything 
+goes wrong.
+
+The process is less complex than the alternative 


This sentence is a little grammatically ambiguous. It is not clear what the preferred method actually is.

hopefully clearer now!

heartsucker · 2019-02-22T09:07:50Z

docs/upgrade/xenial_upgrade_in_place.rst

+
+.. caution::
+  We have tested the upgrade-in-place procedure on officially supported hardware, 
+  but cannot guarantee that the Ubuntu ``do-release-upgrade`` command will 


This is the first reference to the command do-release-upgrade. An admin may not understand this. It should either be referenced in the introductory paragraphs or we should just use a phrase "automated upgrade".

emkll

Instructions for in-place upgrade look good to me, have not gone through the backup instructions yet. I'll let others also take a pass at reviewing these docs. I've left a few minor comments inline.

emkll · 2019-02-22T14:47:07Z

docs/upgrade/xenial_backup_install_restore.rst

+ ./securedrop-admin backup
+
+This will create a backup named ``sd-backup-<date>.tar.gz`` in the
+``~/Persistent/securedrop/install_files/ansible-base`` directory. Make a note of


Would it be worth stating that the backups are not encrypted and to be mindful if/when copying them to other devices/storage?

added a note to that effect.

emkll · 2019-02-22T14:49:09Z

docs/upgrade/xenial_backup_install_restore.rst

+
+ cd ~/Persistent/securedrop
+ git checkout 0.12.0
+ git tag -v 0.12.0


We should validate the tag before checking it out, which is consistent with the updater behavior

updated this snippet in both docs.

emkll · 2019-02-22T14:57:25Z

docs/upgrade/xenial_backup_install_restore.rst

+.. code:: sh
+
+ cd ~/Persistent
+ cp app_services/app*ths securedrop/install_files/ansible-base/


Based on the command at L64:

cp securedrop/install_files/ansible-base/app-*ths app_service/

Will this also copy ssh-app-aths?

app_services -> app_service

it's app-ssh-aths iirc.

emkll · 2019-02-22T15:08:40Z

docs/upgrade/xenial_upgrade_in_place.rst

+Disconnect the SSH session to the Application Server. You are now ready to move
+on to the next step: reinstalling SecureDrop on the Xenial servers.
+
+Step 3: Reinstall SecureDrop 


I propose changing the title to either re-provision or re-configure SecureDrop servers

emkll · 2019-02-22T15:10:38Z

docs/upgrade/xenial_upgrade_in_place.rst

+.. code:: sh                                                                    
+
+ cd ~/Persistent/securedrop                                                     
+ git checkout 0.12.0                                                            


Same as above, git fetch --tags might be necessary, and it's more prudent to verify the tag before checking it out, and it's consistent with what's done in the updater as well.

emkll · 2019-02-22T15:13:28Z

docs/upgrade/xenial_upgrade_in_place.rst

+``examplenot4real.onion``, then the address you should visit is
+``examplenot4real.onion/metadata``. That page should show you key/value pairs,
+including ``0.12.0`` for ``sd_version`` and ``16.04`` for ``server_os``.
+


Ensuring OSSEC emails are flowing would be another interesting test to validate the upgrade

added a section for this.

emkll · 2019-02-22T15:18:07Z

docs/upgrade/xenial_upgrade_in_place.rst

+Contact us
+----------
+If you have questions or comments regarding this process, or if you
+encounter any issues, you can always contact us by the following means:


In your experience, how much instance-specific data are in the error messages when upgrades fail? Enough to recommend they be mindful of org-specific data in public forums (forum, github)

emkll · 2019-02-22T15:22:59Z

docs/upgrade/xenial_backup_install_restore.rst

+
+.. code:: sh
+
+  ./securedrop-admin setup


For users that have not updated Tails in a while, there will be the install once/install each time prompt when downloading apt package (described in https://docs.securedrop.org/en/release-0.11.0/install.html). Since either option will work, might not be worth mentioning here?

gonna leave it out to keep the doc clean

emkll · 2019-02-22T15:24:29Z

docs/upgrade/xenial_backup_install_restore.rst

+.. code:: sh
+
+ cd ~/Persistent/securedrop
+ git checkout 0.12.0


git fetch --tags might be necessary here since it's not cloned

- Use caution/warnings blocks in a couple of places; - Clean up trailing whitespace and wrap to 80 characters; - Follow formatting recommendations as closely as possible; - Small wording changes, reduce direct repetition; - Correct final step: install instead of sdconfig.

@conorsch

- Minimal additional tests (most failures should surface in playbook run) and an end-to-end test for good measure - Clarify what output is expected for `lsb_release -a` - Make it clearer how to resize terminal (H/T @conorsch) in case of ncurses issues - Remove placeholder notes

- Try to distinguish more clearly between new/existing hardware - Explicitly call out location of IP address info

updates to Xenial install docs based on review

emkll · 2019-02-25T15:53:28Z

docs/upgrade/xenial_backup_install_restore.rst

+  cd ~/Persistent/
+  mkdir app_service
+  cp securedrop/install_files/ansible-base/app-*ths app_service/
+  cp securedrop/install_files/ansible-base/site-specific app_service/


isn't the path install_files/ansible-base/group_vars/all/site-specific by default?

kushaldas · 2019-02-25T14:35:50Z

docs/upgrade/xenial_backup_install_restore.rst

+  cd ~/Persistent/
+  mkdir app_service
+  cp securedrop/install_files/ansible-base/app-*ths app_service/
+  cp securedrop/install_files/ansible-base/site-specific app_service/


This path is wrong, the actual path is:

securedrop/install_files/ansible-base/group_vars/all/site-specific

kushaldas · 2019-02-25T14:38:27Z

docs/upgrade/xenial_upgrade_in_place.rst

+- Choose **Yes** on the "Configuring libssl.0.0:amd64" dialog - when you do so,
+  the ``sshd`` and ``tor`` daemons will restart, breaking your connection to the
+  *Monitor Server*
+- Close the terminal window, open a new one, and reconnect with the command


There should be a note that if someone is connected over local ssh, then it will not be disconnected.

I haven't tested this scenario, but if sshd is bounced wouldn't they be disconnected?

Never happened to me.

updated tmux note, fixed path error

redshiftzero · 2019-02-26T01:31:46Z

docs/upgrade/xenial_backup_install_restore.rst

+- via our `community forums <https://forum.securedrop.org>`_.
+
+If you encounter problems that are not security-sensitive, we also encourage you
+to `file an issue <https://github.com/freedomofpress/securedrop/issues/new/>`


this link needs a _ after the final backtick (it's not rendered as a link currently)

kushaldas

Followed the steps for upgradation on Intel NUCs. This is good to merge. 👍

eloquence force-pushed the 4057-docs-xenial-update branch 3 times, most recently from ab0ed81 to ee1adba Compare February 20, 2019 02:32

eloquence force-pushed the 4057-docs-xenial-update branch 2 times, most recently from fc1ffbf to 6876a25 Compare February 22, 2019 00:21

eloquence marked this pull request as ready for review February 22, 2019 01:13

eloquence requested review from emkll, kushaldas and redshiftzero as code owners February 22, 2019 01:13

eloquence changed the title ~~Add documentation for Xenial upgrade~~ [xenial] Add documentation for Xenial upgrade Feb 22, 2019

eloquence added this to Ready for review in SecureDrop Team Board Feb 22, 2019

heartsucker reviewed Feb 22, 2019

View reviewed changes

emkll reviewed Feb 22, 2019

View reviewed changes

zenmonkeykstop and others added 8 commits February 22, 2019 09:37

Made docs changes to support Xenial installs and upgrades

8d6fc41

Restore fresh install guide to original state; handled in #4157

c7c522a

added backup/install/restore docs

1243d24

updated upgrade-in-place docs

def5d6c

First pass copyedit/formatting, small tweaks:

4dc3c75

- Try to distinguish more clearly between new/existing hardware - Explicitly call out location of IP address info

Securedrop->SecureDrop

5a01ddb

eloquence force-pushed the 4057-docs-xenial-update branch from 2c8f18e to 5a01ddb Compare February 22, 2019 17:38

eloquence and others added 3 commits February 22, 2019 09:42

Reinstate Xenial-related changes to server docs

66fc3b7

updated based on review

a13f9cc

Merge pull request #4177 from zenmonkeykstop/4057-docs-xenial-update

00f9348

updates to Xenial install docs based on review

eloquence mentioned this pull request Feb 22, 2019

Add 0.11.1 to 0.12.0 upgrade docs #4182

Merged

1 task

emkll reviewed Feb 25, 2019

View reviewed changes

kushaldas requested changes Feb 25, 2019

View reviewed changes

updated tmux note, fixed path error

208108d

Merge pull request #4189 from zenmonkeykstop/4057-docs-xenial-update

e13671d

updated tmux note, fixed path error

redshiftzero reviewed Feb 26, 2019

View reviewed changes

Adds missing _ for links formatting

6d70fb3

kushaldas approved these changes Feb 26, 2019

View reviewed changes

kushaldas merged commit 2bdd7e7 into develop Feb 26, 2019

SecureDrop Team Board automation moved this from Ready for review to Done Feb 26, 2019

kushaldas mentioned this pull request Feb 26, 2019

Docs xenial update backport #4195

Merged

5 tasks

heartsucker deleted the 4057-docs-xenial-update branch February 27, 2019 10:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[xenial] Add documentation for Xenial upgrade #4140

[xenial] Add documentation for Xenial upgrade #4140

eloquence commented Feb 15, 2019 •

edited

codecov-io commented Feb 20, 2019 •

edited

heartsucker left a comment

heartsucker Feb 22, 2019

zenmonkeykstop Feb 22, 2019

heartsucker Feb 22, 2019

zenmonkeykstop Feb 22, 2019

heartsucker Feb 22, 2019

zenmonkeykstop Feb 22, 2019

emkll left a comment

emkll Feb 22, 2019

zenmonkeykstop Feb 22, 2019

emkll Feb 22, 2019

zenmonkeykstop Feb 22, 2019

emkll Feb 22, 2019

zenmonkeykstop Feb 22, 2019

zenmonkeykstop Feb 22, 2019

emkll Feb 22, 2019

zenmonkeykstop Feb 22, 2019

emkll Feb 22, 2019

emkll Feb 22, 2019

zenmonkeykstop Feb 22, 2019

emkll Feb 22, 2019

emkll Feb 22, 2019

zenmonkeykstop Feb 22, 2019

emkll Feb 22, 2019 •

edited

emkll Feb 25, 2019

kushaldas Feb 25, 2019

kushaldas Feb 25, 2019

zenmonkeykstop Feb 25, 2019

kushaldas Feb 26, 2019

redshiftzero Feb 26, 2019

kushaldas left a comment

[xenial] Add documentation for Xenial upgrade #4140

[xenial] Add documentation for Xenial upgrade #4140

Conversation

eloquence commented Feb 15, 2019 • edited

How to test

Status

codecov-io commented Feb 20, 2019 • edited

Codecov Report

heartsucker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emkll left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emkll Feb 22, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kushaldas left a comment

Choose a reason for hiding this comment

eloquence commented Feb 15, 2019 •

edited

codecov-io commented Feb 20, 2019 •

edited

emkll Feb 22, 2019 •

edited