Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[xenial] Perform timeboxed upgrade attempt of SecureDrop from Ubuntu 14.04 to 16.04 #3491

Closed
eloquence opened this issue May 30, 2018 · 7 comments

Comments

Projects
None yet
3 participants
@eloquence
Copy link
Contributor

commented May 30, 2018

As part of the xenial epic ( #3207), similar to a from-scratch install of SecureDrop with Ubuntu 16.04 (#3207), we should attempt the upgrade of an existing SecureDrop instance from 14.04 to 16.04 in a non-repeatable manner, to gather information that will inform our decision about a potential base OS transition.

For the purposes of the 5/30-6/13 sprint, this task is timeboxed to approximately one working day.

@eloquence eloquence added this to Current Sprint Backlog - 5/16 - 5/30 in SecureDrop Team Board May 30, 2018

@conorsch conorsch moved this from Current Sprint Backlog - 5/16 - 5/30 to In Development in SecureDrop Team Board Jun 1, 2018

@conorsch

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2018

Performed in-place upgrade of mon-prod VM from Trusty to Xenial. Results below.

Prerequisites

The do-release-upgrade command showed no pending updates:

vagrant@mon-prod:~$ sudo do-release-upgrade
Checking for a new Ubuntu release
No new release found

Based on the Xenial release notes, it was necessary to edit /etc/update-manager/release-upgrades to change Prompt=never to Prompt=lts. Then do-release-upgrade displayed the Xenial option.

Detailed prompts

Note that the cron-apt logic automatically removes unused packages, so sticking with the default of not removing the obsolete packages is still sane.

Kicking off the upgrade
Do you want to start the upgrade?

12 installed packages are no longer supported by Canonical. You can
still get support from the community.

10 packages are going to be removed. 144 new packages are going to be
installed. 419 packages are going to be upgraded.

You have to download a total of 235 M. This download will take about
2 minutes with your connection.

Installing the upgrade can take several hours. Once the download has
finished, the process cannot be canceled.

Continue [yN]  Details [d] d

No longer supported: biosdevname gcc-4.8-base gcc-4.9-base 
  libarchive-extract-perl liblog-message-simple-perl 
  libmodule-pluggable-perl libpod-latex-perl libterm-ui-perl 
  libtext-soundex-perl module-init-tools python-debian w3m 

Remove: libasprintf0c2 libpython3.4-minimal libpython3.4-stdlib 
  perl-modules python3.4 python3.4-minimal systemd-services 

Remove (was auto installed) aptitude libkyotocabinet16 libxapian22 

Install: cgmanager console-setup-linux cpp-5 distro-info-data 
  dmeventd fontconfig-config fonts-dejavu-core gcc-5 gcc-5-base 
  gcc-6-base init initramfs-tools-core libapt-inst2.0 libapt-pkg5.0 
  libasan2 libasprintf0v5 libbind9-140 libboost-iostreams1.58.0 
  libcc1-0 libcilkrts5 libdns-export162 libdns162 libdrm-amdgpu1 
  libdrm-common libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 
  libfdisk1 libfontconfig1 libfontenc1 libgcc-5-dev libgcrypt20 
  libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa libgnutls30 
  libhogweed4 libice6 libicu55 libisc-export160 libisc160 libisccc140 
  libisccfg140 libisl15 libkyotocabinet16v5 libllvm5.0 
  liblog-message-perl liblsan0 liblvm2app2.2 liblvm2cmd2.02 
  liblwres141 liblz4-1 libmnl0 libmodule-runtime-perl libmpx0 
  libmysqlclient20 libnettle6 libparams-classify-perl libparted2 
  libpciaccess0 libperl5.22 libplymouth4 libprocps4 libpython3.5 
  libpython3.5-minimal libpython3.5-stdlib librtmp1 libsensors4 
  libsm6 libsmartcols1 libsystemd0 libtk8.6 libtxc-dxtn-s2tc0 
  libubsan0 libutempter0 libx11-6 libx11-data libx11-xcb1 
  libxapian-1.3-5 libxapian22v5 libxaw7 libxcb-dri2-0 libxcb-dri3-0 
  libxcb-glx0 libxcb-present0 libxcb-shape0 libxcb-sync1 libxcb1 
  libxcomposite1 libxdamage1 libxext6 libxfixes3 libxft2 libxi6 
  libxinerama1 libxmu6 libxmuu1 libxpm4 libxrandr2 libxrender1 
  libxshmfence1 libxss1 libxt6 libxtables11 libxtst6 libxv1 
  libxxf86dga1 libxxf86vm1 linux-base pastebinit perl-modules-5.22 
  python-attr python-cffi-backend python-cryptography python-enum34 
  python-idna python-ipaddress python-ndg-httpsclient python-pyasn1 
  python-pyasn1-modules python-service-identity python3-chardet 
  python3-debian python3-pkg-resources python3-requests python3-six 
  python3-systemd python3-urllib3 python3-xapian1.3 python3.5 
  python3.5-minimal rename s-nail systemd systemd-sysv tcl-expect 
  tcl8.6 tk8.6 update-motd x11-common x11-utils xbitmaps 
  xdg-user-dirs xterm 
SSH-related config changes
Configuration file '/etc/ssh/moduli'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** moduli (Y/I/N/O/D/Z) [default=N] ? 

Configuration file '/etc/ssh/ssh_config'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** ssh_config (Y/I/N/O/D/Z) [default=N] ? 

Configuration file '/etc/pam.d/sshd'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** sshd (Y/I/N/O/D/Z) [default=N] ? 
Interactive prompts

mon-xenial-upgrade-1-postfix

mon-xenial-upgrade-2-tor-restart

mon-xenial-upgrade-3-grub

mon-xenial-upgrade-4-grub-partitions

mon-xenial-upgrade-5-unattended-upgrades

Obsolete packages
Remove obsolete packages? 

29 packages are going to be removed. 

 Continue [yN]  Details [d]d
Remove (was auto installed) apt-xapian-index aptitude-common cpp-4.8 
  gcc-4.8 heirloom-mailx libasan0 libboost-iostreams1.54.0 
  libboost-iostreams1.58.0 libck-connector0 libclass-accessor-perl 
  libcloog-isl4 libcwidget3 libept1.4.12 libgcc-4.8-dev libgssglue1 
  libisl10 libmysqlclient18 libparse-debianchangelog-perl 
  libsigc++-2.0-0c2a libsub-name-perl libtimedate-perl 
  libxapian-1.3-5 libxapian22v5 python-ndg-httpsclient 
  python-requests python-urllib3 python-xapian python3-xapian1.3 
  watershed 
Final reboot
System upgrade is complete.

Restart required 

To finish the upgrade, a restart is required. 
If you select 'y' the system will be restarted. 

Continue [yN] y

Then, after a reboot:

vagrant@mon-prod:~$ uname -a
Linux mon-prod 4.4.115-grsec #1 SMP Mon Feb 5 15:01:26 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
vagrant@mon-prod:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:        16.04
Codename:       xenial

Fixes

Had problems with DNS resolution via apt post-upgrade. Updated the firewall rules to authorize the _apt user, rather than root, to call out over HTTP/S ports. This resolved the problem, and sudo apt-get update worked well again. There may be additional DNS/NTP-related troubleshooting required, given the change to systemd-resolved.

Summary

Throughout the install, default choices were accepted as sufficient, with the following exceptions:

  1. The grub partition selection required manual selection of the appropriate partition. The choices displayed above are VM-specific, so we'll need to evaluate the choices presented by the upgrader on hardware.
  2. The service bounce disconnected tor, breaking the SSH connection, so we should consider not restarting tor as part of the openssl changes. Reconnecting over SSH in a new terminal window resumed the upgrade operation, after tor bounce, given the tmux attach logic.
  3. The final reboot prompt defaults to "no", so overriding and choosing "yes" is required for immediate reboot.

Additionally, firewall tweaks were required. If we move to managing system state in an unattended fashion, e.g. via ansible-pull as described in #3136, we can ensure the firewall rules are correctly set. We should critically evaluate the uid restrictions, given the complications for smooth upgrades.

Will proceed with in-place upgrade testing of App VMs, and report back.

@eloquence

This comment has been minimized.

Copy link
Contributor Author

commented Jun 12, 2018

Since this was a timeboxed research task, if there are no objections, I suggest adding any final notes and closing this task before the beginning of the next sprint.

@conorsch

This comment has been minimized.

Copy link
Contributor

commented Jun 12, 2018

Upgrade testing on app-staging is going smoothly. Here's the WIP branch, showing the diff: https://github.com/freedomofpress/securedrop/compare/test-xenial-upgrade-path

The testing procedure is as follows:

  1. Checkout develop branch.
  2. make build-debs && vagrant provision /stag/
  3. Check out test-xenial-upgrade-path branch.
  4. make build-debs && ANSIBLE_ARGS="--tags rebuild" vagrant provision /stag/
  5. vagrant ssh app-staging
  6. sudo do-release-upgrade Follow prompts, accept defaults (see notes above for screenshots).
  7. Reboot

Voila!

vagrant@app-staging:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:        16.04
Codename:       xenial
vagrant@app-staging:~$ uname -r
4.4.115-grsec
vagrant@app-staging:~$

app-staging-xenial-source-interface

All in all, an attended upgrade from Trusty -> Xenial seems feasible. Still worth considering the approach for more control over system state described in #3136, but so far the relatively small number of changes can be managed via postinst if necessary.

@conorsch conorsch closed this Jun 12, 2018

SecureDrop Team Board automation moved this from In Development to Done Jun 12, 2018

@conorsch

This comment has been minimized.

Copy link
Contributor

commented Jun 12, 2018

Additions from notes that may be relevant down the road:

  • haveged apparmor profile may not be necessary, that was generated from aa-logprof. We should try removing it to see if it's really necessary.
  • The PAM common-auth customizations include declarations for pam_ecryptfs.so; I had to comment that line out. Did not investigate thoroughly.

That info should be enough to recreate the work above if we decide to move forward with a Trusty to Xenial migration. See also the Ansible logic changes, of course, in #3207.

@kushaldas

This comment has been minimized.

Copy link
Contributor

commented Dec 3, 2018

trusty-xenial1

I got this on the app server.

@kushaldas

This comment has been minimized.

Copy link
Contributor

commented Dec 3, 2018

trusty-xenial2

Then this one, I chose No

@kushaldas

This comment has been minimized.

Copy link
Contributor

commented Dec 3, 2018

trusty-xenial3
trusty-xenial4
trusty-xenial5
trusty-xenial6
screenshot from 2018-12-04 02-22-23

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.