cron-apt remove action should be after security

[emkll]

Status

Ready for review

Description of Changes

Fixes #4003 .

In the event the host has broken packages installed (e.g. failed postinst), cron-apt will fail at the remove action before upgrading the package. Upgrades to the broken package will not be installed and the system will remain in a broken state:

Change 1-remove to 9-remove in cron-apt actions for Ansible install as well as in securedrop-config postinst script (and delete 1-remove via the same)

Testing

Clean install scenario:

• Successfully provision staging or prod environment on this branch
• /etc/cron-apt/action.d/ contains files as expected (0-update 5-security and 9-remove)
• sudo cron-apt -i -s completes without error

Upgrade scenario

Using (https://docs.securedrop.org/en/release-0.10.0/development/upgrade_testing.html):

• Checkout this branch
• Install SecureDrop <=0.10.0:
  • make build-debs
  • molecule converge -s upgrade
  • copy and install securedrop-config deb package produced by this branch
• Insert duplicate entry for mon in /etc/hosts (either app or mon)
• Attempt an upgrade to 0.11.0:
  • /etc/apt/security.list should contain apt.freedom.press
  • run cron-apt -i -s and observe ossec failing to install
• Attempt an upgrade to 0.12~rc1:
  • /etc/apt/security.list should contain apt-test.freedom.press
  • run cron-apt -i -s and observe ossec failing to install

• Install of ossec fails, but other packages are still upgraded (e.g. apt list --installed | grep securedrop-app-code should return that version of securedrop-app-code installed is 0.12.0~rc1

Deployment

• Existing installs will be updated via securedrop-config apt package
• New installs will be provisioned by the correct config via Ansible

Checklist

If you made changes to the system configuration:

• Configuration tests pass

If you made non-trivial code changes:

• I have written a test plan and validated it for this PR

[zenmonkeykstop]

Tested fresh install scenario on libvirt staging VMS on Ubuntu 18.04 host:

[x] make build-debs crashed out at test phase because of build container dependency check, but debs built, but provisioning went fine otherwise.
[x] /etc/cron-apt/action.d/ contained files as expected (0-update 5-security and 9-remove)
[x] sudo cron-apt -i -s completed without error

[conorsch]

Proceeding with review of the upgrade scenario. Using a slightly modified test plan from what's in the OP, so documenting here for clarity:

Upgrade to 0.9.1 -> 0.11.0 -> 0.12.0~rc1

• Check out this branch
• make build-debs (so the local debs used by the upgrade scenario include the changes presented here)
• echo "0.9.1" > molecule/shared/stable.ver (to force use of 0.9.1 boxes as upgrade target)
• make upgrade_start (to prep boxes on version 0.9.1)
• Confirm "0.9.1" version string on Source Interface
• Insert duplicate entry for mon in /etc/hosts on app-staging host
• Run sudo apt-get install --only-upgrade securedrop-config (to get latest deb from this branch, via local apt)
• Comment out override for apt.freedom.press in /etc/hosts on app-staging host (to use prod, not local, apt server)
• Run sudo rm -r /var/lib/apt/lists to purge local apt versions from apt cache
• Manually run sudo cron-apt -i -s inside app-staging host (to pull 0.11.0 packages via prod apt)
• Observe failure on securedrop-ossec-agent package upgrade in cron-apt output
• Confirm "0.11.0" version string on Source Interface (:tada:!)
• Re-run sudo rm -r /var/lib/apt/lists to purge prod apt versions from apt cache
• Re-enable override for apt.freedom.press in /etc/hosts on app-staging host (to use local, not prod, apt server)
• Re-run sudo cron-apt -i -s inside app-staging host (to pull 0.12.0~rc1 packages via local apt)
• Observe failure on securedrop-ossec-agent package upgrade in cron-apt output again
• Confirm "0.12.0~rc1" version string on Source Interface (:tada:!)

[conorsch]

Works as advertised.

The changes here will not be applied to any instance currently experiencing upgrade problems—manual intervention is still required there—but the logic here will avoid similar problems in the future.