Xenial transition planning 2018 12 03

SecureDrop: Ubuntu Xenial transition planning, December 3, 2018

Timeline

Will have potentially two server versions in prod at the same time -- then after EOL, Trusty may no longer get app updates

May require custom tooling to run upgrade via Admin Workstation

securedrop-admin tooling could also cover logging
scope of existing tests (which assert that kernel package is correct, etc.) may not be sufficient

Admins will need to enable update, run do-release-upgrade.

Problem: big changes from one Ubuntu release to another could leave server in inconsistent state, e.g. initd to systemd transition
More research required on whether this is likely to arise in practice

Staggered upgrade w/ on-site assistance

How much staff time dedicated -- who will be doing it?
On-site will be done:
- for first set of instances
- for clients that are default "high touch"
- for reported customizations by priority client that may need this level of handholding to untangle (balance vs. reinstall)

Upgrade vs. Reinstall

"Inconsistent state" issue
Main issue is between instances that have security upgrades only vs. all packages upgraded -- Manual upgrades -- Ansible playbook run: changing cert, changing OSSEC key, adding languages -> When package is held back, this may not be visible to us
SecureDrops beyond our realistic reach

Do we need a system state snapshot from running instances?

list of packages/versions installed (and whether they were manually installed or pulled in as a dependency)
OSSEC logs? not as useful due to volume of upgrades Preliminary conclusion: get logs on-demand, but don't request huge batch of logs that we don't have bandwidth to sift through

To-dos:

Reorganizing playbooks to support two instead of one
Reduce level of custom work (incrementally?)
Might make sense to have CI staging job run for both Trusty and Xenial for some time --> This should be easier with GCE
apt-test server needs to support xenial channel -- deploy logic needs to respect channel
investigate noninteractive do-release-upgrade
test Backup on Trusty -> restore on Xenial
fix issues identified in: https://docs.securedrop.org/en/release-0.10.0/development/xenial_support.html#known-bugs-with-xenial-support
add package/version list to securedrop-admin logs
prepare bulk messaging to admins
messaging on journalist interface
message through securedrop message submission
plan out assisted migrations for priority clients
compare clean xenial install state vs. trusty->xenial post-upgrade state

Jen/Conor/Erik to map out issues, timeline

Provide feedback