[xenial] Ensures server packages required by ./securedrop-admin install are present on Xenial

[zenmonkeykstop]

Status

Ready for review

Description of Changes

Fixes #4109 .

Adds an Ansible role that ensures securedrop-admin install's missing package prerequisites (python, apt-transport-https, dnsutils) are installed on Xenial systems set up with only the OpenSSH server option checked in the OS installer's 'Software selection` dialog.

Testing

Prod VMs

• Modify Vagrantfile if necessary to use 16.04 images for app-prod and mon-prod
• Create prod VMs: vagrant up --no-provision /prod/
• Use vagrant ssh <vm-name-here to connect to VMs, and remove the packages listed above. Verify that /usr/bin/python in particular has been removed.
• On your Tails VM Admin Workstation, modify /Persistent/securedrop/install_files/ansible-base/roles/install-fpf-repo/defaults/main.yml to point to the apt-test.freedom.press test repo and to use apt-test-signing-key.pub as the repo signing key
• proceed with the SecureDrop installation

Hardware

• Install Xenial on app and mon servers with the usual options, ensuring that standard system utilities is deselected and OpenSSH server is selected on the Software Selection dialog.
• On your Admin Workstation, modify ~/Persistent/securedrop/install_files/ansible-base/roles/install-fpf-repo/defaults/main.yml to point to the apt-test.freedom.press test repo and to use apt-test-signing-key.pub as the repo signing key
• Proceed with installation as normal.

Deployment

This only affects new Xenial installs, and nothing special is required for deployment.

Checklist

If you made non-trivial code changes:

• I have written a test plan and validated it for this PR
  (note: tested on prod VMs)

[kushaldas]

The change looks good in my mind 👍

I could not test it as my tails vms are not taking any keyboard input :(

[kushaldas]

@zenmonkeykstop This also needs a rebase against the develop to fix the mypy issue.

[emkll]

If the apt package manager on the host is vulnerable to CVE-2019-3462 (which is likely currenly the case, since the latest Xenial ISO is 6 months old [0]), it may be possible for an attacker to exploit this vulnerability and run malicious code. The upgrade-apt.yml task introduced in #4061 unfortunately runs after the prepare-servers role specified here.

It might make sense to convert the apt upgrade logic into the prepare-servers task in raw commands, to ensure it is consistently run before everything else.

[0] : http://releases.ubuntu.com/16.04/

[conorsch]

Appended a commit moving the logic already present via #4061 into the new role.

The outstanding question I have now is: will the raw apt-get task work against fresh hardware hosts? Recall that the VM images we use for testing already have passwordless sudo enabled. Given the sudo call in the raw task, coupled with the fact that we haven't yet enabled passwordless sudo as part of the install flow, the task may block indefinitely on a password prompt. That would be an inscrutable error for Admins to face as part of the first install experience.

Hardware testing required in order to confirm the above. If it does prove to be a problem, then we're faced with a few options:

1. Conditionally run the task if it's reasonable to assume this is a first time install (e.g. ansible_host does not end in .onion)
2. Attempt to safely pass in sudo password since it was collected via securedrop-admin install ; prefer to avoid, given that we'd be reinventing Ansible wheels doing so
3. Update the install docs to instruct admins to install python2 manually, as part of the pubkey bootstrapping
4. Optional Contingent on 3); we could fail if python2 is not installed, and point Admins to the docs to reduce confusion if the manual install step is overlooked

Slight preference for 3 and optionally 4, but defer to the team for input.

[zenmonkeykstop]

Apologies @conorsch I think I may have nuked your commit (albeit with one doing something similar.)

I won't be in a position to test against hardware until Saturday, if someone else can test against hardware I'd appreciate it. I'm optimistic that it will work if raw commands use a similar mechanism to gather_facts.

[conorsch]

@zenmonkeykstop Thanks for reply, no surprise we're stepping on toes a bit with this one. 😇 Just added some more fixes on top of yours, pulling in some work that was done by @emkll in tandem. I believe this is now ready for review. cc @redshiftzero

I remain skeptical about the sudo password behavior, but testing on hardware required to get clarity there.

[kushaldas]

Failed with the following error while installing from the tails vm.

TASK [install-fpf-repo : Setup FPF apt repo.] ******************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: apt.cache.FetchFailedException: W:The repository 'https://apt.freedom.press xenial Release' does not have a Release file., W:Data from such a repository can't be authenticated and is therefore potentially dangerous to use., W:See apt-secure(8) manpage for repository creation and user configuration details., E:Failed to fetch https://apt.freedom.press/dists/xenial/main/binary-amd64/Packages  404  Not Found, E:Some index files failed to download. They have been ignored, or old ones used instead.
fatal: [mon]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_zTOCpM/ansible_module_apt_repository.py\", line 550, in <module>\n    main()\n  File \"/tmp/ansible_zTOCpM/ansible_module_apt_repository.py\", line 542, in main\n    cache.update()\n  File \"/usr/lib/python2.7/dist-packages/apt/cache.py\", line 487, in update\n    raise FetchFailedException(e)\napt.cache.FetchFailedException: W:The repository 'https://apt.freedom.press xenial Release' does not have a Release file., W:Data from such a repository can't be authenticated and is therefore potentially dangerous to use., W:See apt-secure(8) manpage for repository creation and user configuration details., E:Failed to fetch https://apt.freedom.press/dists/xenial/main/binary-amd64/Packages  404  Not Found, E:Some index files failed to download. They have been ignored, or old ones used instead.\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: apt.cache.FetchFailedException: W:The repository 'https://apt.freedom.press xenial Release' does not have a Release file., W:Data from such a repository can't be authenticated and is therefore potentially dangerous to use., W:See apt-secure(8) manpage for repository creation and user configuration details., E:Failed to fetch https://apt.freedom.press/dists/xenial/main/binary-amd64/Packages  404  Not Found, E:Some index files failed to download. They have been ignored, or old ones used instead.
fatal: [app]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_J_kKjC/ansible_module_apt_repository.py\", line 550, in <module>\n    main()\n  File \"/tmp/ansible_J_kKjC/ansible_module_apt_repository.py\", line 542, in main\n    cache.update()\n  File \"/usr/lib/python2.7/dist-packages/apt/cache.py\", line 487, in update\n    raise FetchFailedException(e)\napt.cache.FetchFailedException: W:The repository 'https://apt.freedom.press xenial Release' does not have a Release file., W:Data from such a repository can't be authenticated and is therefore potentially dangerous to use., W:See apt-secure(8) manpage for repository creation and user configuration details., E:Failed to fetch https://apt.freedom.press/dists/xenial/main/binary-amd64/Packages  404  Not Found, E:Some index files failed to download. They have been ignored, or old ones used instead.\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}

NO MORE HOSTS LEFT *********************************************************************************************

NO MORE HOSTS LEFT *********************************************************************************************
	to retry, use: --limit @/home/amnesia/Persistent/securedrop/install_files/ansible-base/securedrop-prod.retry

PLAY RECAP *****************************************************************************************************
app                        : ok=8    changed=3    unreachable=0    failed=1   
localhost                  : ok=4    changed=0    unreachable=0    failed=0   
mon                        : ok=8    changed=3    unreachable=0    failed=1   

ERROR (run with -v for more): Command '['/home/amnesia/Persistent/securedrop/install_files/ansible-base/securedrop-prod.yml', '--ask-become-pass']' returned non-zero exit status 2

[emkll]

Hey @kushaldas it seems like you are using the prod apt server (apt.freedom.press) to test under xenial. There is currently no Xenial channel hosted on that server: You must use apt-test.freedom.press. The Xenial channel will be hosted on prod apt servers starting with the 0.12.0 release

Dismissing based on recent changes to the branch

[kushaldas]

Hey @kushaldas it seems like you are using the prod apt server (apt.freedom.press) to test under xenial. There is currently no Xenial channel hosted on that server: You must use apt-test.freedom.press. The Xenial channel will be hosted on prod apt servers starting with the 0.12.0 release

@emkll can you please add the steps for the same in the test plan? This will help in future reference.

[zenmonkeykstop]

Apologies for the omission @kushaldas, I've updated the test plan.

[redshiftzero]

✅ stamping after a successful fresh Xenial install on hardware (Mac Minis) where only OpenSSH was enabled during the Ubuntu install.

[redshiftzero]

@kushaldas please take another look at this one when you get a chance today, let's try to get this merged in for 0.12.0~rc2 which we should get done before the weekend

[kushaldas]

• Modify Vagrantfile if necessary to use 16.04 images for app-prod and mon-prod
• Create prod VMs: vagrant up --no-provision /prod/
• Use vagrant ssh <vm-name-here to connect to VMs, and remove the packages listed above. Verify that /usr/bin/python in particular has been removed.
• On your Tails VM Admin Workstation, modify /Persistent/securedrop/install_files/ansible-base/roles/install-fpf-repo/defaults/main.yml to point to the apt-test.freedom.press test repo and to use apt-test-signing-key.pub as the repo signing key
• proceed with the SecureDrop installation and it finished and worked as expected.