Updates developer docs for upgrade testing targets

[conorsch]

Status

Ready for review / Work in progress

Description of Changes

Follow-up to #3819. Changes proposed in this pull request:

• Updates developer documentation to use new Makefile targets
• Tweaks make upgrade-test-qa target to force use of apt-test proxy

It's now possible to run full upgrade testing, end to end, with two commands:

• make upgrade-start
• make upgrade-test-qa

Pushed up new debs to apt-test, built from 71ca2ed (the latest develop at time of writing).

Also closes #3998, by documenting the maintenance procedures around preparing new boxes.

Testing

Read the new docs and review for clarity. @msheiny in particular should sanity check the new make upgrade-test-qa logic, but it seems reasonable to me.

Deployment

Any special considerations for deployment? Consider both:

1. Upgrading existing production instances.
2. New installs.

Checklist

If you made changes to the server application code:

• Linting (make ci-lint) and tests (make -C securedrop test) pass in the development container

If you made changes to securedrop-admin:

• Linting and tests (make -C admin test) pass in the admin development container

If you made changes to the system configuration:

• Configuration tests pass

If you made non-trivial code changes:

• I have written a test plan and validated it for this PR

If you made changes to documentation:

• Doc linting (make docs-lint) passed locally

[codecov-io]

Codecov Report

Merging #3832 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           develop   #3832   +/-   ##
=======================================
  Coverage     84.7%   84.7%           
=======================================
  Files           43      43           
  Lines         2765    2765           
  Branches       300     300           
=======================================
  Hits          2342    2342           
  Misses         355     355           
  Partials        68      68

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 985c59c...cadd878. Read the comment docs.

[zenmonkeykstop]

Looks good, with the exception of one point re: upgrades using apt-test.

[zenmonkeykstop]

This step will fail unless there are previously-built Debian packages for Molecule to copy to the apt server and sign. The "Copy over debs" task succeeds even though build/ is empty, but "Sign release file" fails. Since the packages aren't actually needed when upgrading from apt-test, maybe the "Sign release file" task should just check for the existence of a Release file first and do nothing if it doesn't exist.

[msheiny]

Yeppppp @zenmonkeykstop is totally right here.. that logic be busted :| Ill try to push a fix for that real quick. I like how quickly the tides of this PR changes from nits+docs --> bugfixes. ❤️

[conorsch]

This step will fail unless there are previously-built Debian packages for Molecule to copy to the apt server

Just ran into this myself in a clean working directory. We can pilfer the find-debs logic used in the staging logic and fail fast if the build/ directory is empty. Will append a commit.

[msheiny]

Currently failing install in two diff scenarios:

1 - The change to upgrade-test-qa target is breaking because you need to pass SD_UPGRADE_BASE to any converge. I'll push a fix for that.. easy peasy

2 - The issue @zenmonkeykstop brings up seems to be an existing issue... this opens up a can of worms. I was mid opening a ticket for this but I realized the root of the issue is the divergence of workflow between the local dpkg and remote qa workflows is dramatically different.. maybe we should just create two sets of start make commands here?

[msheiny]

Found a bug here.. fixing with an incoming commit

[msheiny]

Yeppppp @zenmonkeykstop is totally right here.. that logic be busted :| Ill try to push a fix for that real quick. I like how quickly the tides of this PR changes from nits+docs --> bugfixes. ❤️

[msheiny]

i was about to push a fix but then realized this opens a discussion here that needs addressing...

to break it down - there is a divergence between the local dpkg and remote apt-test testing scenarios that is showing its head here. Basically if you are going to use the remote apt-test scenario... you should pass that in the beginning (at the initial converge stage) and not afterwards as this doc change now reflects.

The method now described in the docs opens up a bug as @zenmonkeykstop pointed out - specifically when the build dir is empty. So the quickest fix is to add a parallel make target something like make upgrade-start-qa and have that reflected in the docs.

I was about to push this fix:

diff --git a/Makefile b/Makefile
index 6b6fd3b1d..a0ba33530 100644
--- a/Makefile
+++ b/Makefile
@@ -171,6 +171,10 @@ clean: ## DANGER! Purges all site-specific info and developer files from project
 upgrade-start: ## Boot up an upgrade test base environment using libvirt
        @SD_UPGRADE_BASE=$(STABLE_VER) molecule converge -s upgrade

+.PHONY: upgrade-start-qa
+upgrade-start-qa: ## Boot up an upgrade test base env using libvirt in remote apt mode
+       @SD_UPGRADE_BASE=$(STABLE_VER) QA_APTTEST=yes molecule converge -s upgrade
+
 .PHONY: upgrade-destroy
 upgrade-destroy: ## Destroy up an upgrade test base environment
        @SD_UPGRADE_BASE=$(STABLE_VER) molecule destroy -s upgrade

The issue with the currently documented change to the make target of upgrade-test-qa is that it will fall over because there is no SD_UPGRADE_BASE env var being passed as-is. That is also an easy fix tho.

[msheiny]

The method now described in the docs opens up a bug as @zenmonkeykstop pointed out

Sorry to clarify - that bug was already present thanks to yours truly 👍 @zenmonkeykstop just pointed out a short-coming I made by not adding a parallel QA make target to address that bug.

Changes implemented via provided patch

[conorsch]

Ready for review. Implemented the changes flagged by both @msheiny and @zenmonkeykstop . The most recent docs additions also close #3998. Note that I did not save a box for v0.10.0 of SD (see commit message for details), but we now have a 0.11.0 box live to aid in upgrade testing for 0.12.0.

[zenmonkeykstop]

This is consistently failing for me with the 0.11.0 boxes, at the make upgrade-start command, during the "Create molecule instances" task. The task times out during the mon VM setup. Looking at the console in virt-manager, I see filesystem errors on boot (screenshot attached). Is it possible the box is busted?

[conorsch]

That's no fun. Thanks for report, @zenmonkeykstop — I will purge my local boxes and pull down fresh to see if I can reproduce.

[zenmonkeykstop]

No problem - fwiw I updated the version in molecule/shared/stable.ver to 0.10.0 and Molecule sucked down the 0.10.0 boxes and completed without error (though the application version installed was 0.11.0 which seems a little confusing).

[conorsch]

Molecule sucked down the 0.10.0 boxes and completed without error (though the application version installed was 0.11.0 which seems a little confusing).

That's a bug, I had intended to remove the 0.10.0 metadata references (which would cause fetching 0.10.0 boxes to fail) since we missed building boxes for 0.10.0 during the release, and manually cobbling together a box for 0.10.0 bumped into the timebox I'd set. Will follow up excising that.

[conorsch]

@zenmonkeykstop please try again. I've removed the metadata reference to the 0.10.0 box, since it had 0.11.0 installed, as you pointed out. I did a rebase, squash, and force push, because I'd already written a commit message indicating that 0.10.0 wasn't include—that message is now accurate!

As for the 0.11.0 failure, I'm unable to reproduce that problem. After purging the 0.11.0 boxes from my host, I'm able to run make upgrade-start just fine and have it complete without error. Can you try again? Here's how I purged the boxes to test again:

vagrant box remove fpf/securedrop-app --box-version 0.11.0
vagrant box remove fpf/securedrop-mon --box-version 0.11.0

Note that you may need to cd ~/ first in order to sidestep Vagrant version mismatches.

[zenmonkeykstop]

I purged boxes as above and saw the same error, but deleting the images in question from /var/lib/libvirtd/images and restarting libvirtd did the trick! (There's almost certainly a cleaner way to do this with virsh.) So I'm inclined to chalk this up to getting a dodgy version of the 0.11.0 mon box at some point.

With fresh boxes, local and apt-test upgrades work as expected. LGTM!

[conorsch]

Thanks for re-review, @zenmonkeykstop! My mistake on recommending the vagrant box remove command, glad you were able to sort it anyway.

@kushaldas Can you take a spin through these docs and add your review? If you approve, as well, then it's good to go in. We may also be able to close #4025 in tandem, as well.

[kushaldas]

While trying to execute the command: make upgrade-start I got the following error.

    TASK [Gathering Facts] *********************************************************
    ok: [localhost]
    
    TASK [Create molecule instance(s)] *********************************************
    failed: [localhost] (item={'box': u'fpf/securedrop-app', 'name': u'app-staging', 'provider_override_args': [u"vm.synced_folder './', '/vagrant', disabled: true, type: 'nfs'"], 'box_url': u'../vagrant_packager/box_files/app_metadata.json', 'private_ip': u'10.0.1.2', 'groups': [u'securedrop_application_server', u'securedrop', u'staging'], 'memory': 1024, 'instance_raw_config_args': [u'ssh.insert_key = false']}) => {"changed": false, "item": {"box": "fpf/securedrop-app", "box_url": "../vagrant_packager/box_files/app_metadata.json", "groups": ["securedrop_application_server", "securedrop", "staging"], "instance_raw_config_args": ["ssh.insert_key = false"], "memory": 1024, "name": "app-staging", "private_ip": "10.0.1.2", "provider_override_args": ["vm.synced_folder './', '/vagrant', disabled: true, type: 'nfs'"]}, "msg": "ERROR: See log file '/tmp/molecule/securedrop/upgrade/vagrant-app-staging.err'"}

It is still stuck at this error, so I am waiting to see what will happen next.

[kushaldas]

The latest try failed at this output.

    TASK [Create molecule instance(s)] *********************************************
    failed: [localhost] (item={'box': u'fpf/securedrop-app', 'name': u'app-staging', 'provider_override_args': [u"vm.synced_folder './', '/vagrant', disabled: true, type: 'nfs'"], 'box_url': u'../vagrant_packager/box_files/app_metadata.json', 'private_ip': u'10.0.1.2', 'groups': [u'securedrop_application_server', u'securedrop', u'staging'], 'memory': 1024, 'instance_raw_config_args': [u'ssh.insert_key = false']}) => {"changed": false, "item": {"box": "fpf/securedrop-app", "box_url": "../vagrant_packager/box_files/app_metadata.json", "groups": ["securedrop_application_server", "securedrop", "staging"], "instance_raw_config_args": ["ssh.insert_key = false"], "memory": 1024, "name": "app-staging", "private_ip": "10.0.1.2", "provider_override_args": ["vm.synced_folder './', '/vagrant', disabled: true, type: 'nfs'"]}, "msg": "ERROR: See log file '/tmp/molecule/securedrop/upgrade/vagrant-app-staging.err'"}
    failed: [localhost] (item={'box': u'fpf/securedrop-mon', 'name': u'mon-staging', 'provider_override_args': [u"vm.synced_folder './', '/vagrant', disabled: true, type: 'nfs'"], 'box_url': u'../vagrant_packager/box_files/mon_metadata.json', 'private_ip': u'10.0.1.3', 'groups': [u'securedrop_monitor_server', u'securedrop', u'staging'], 'memory': 1024, 'instance_raw_config_args': [u'ssh.insert_key = false']}) => {"changed": false, "item": {"box": "fpf/securedrop-mon", "box_url": "../vagrant_packager/box_files/mon_metadata.json", "groups": ["securedrop_monitor_server", "securedrop", "staging"], "instance_raw_config_args": ["ssh.insert_key = false"], "memory": 1024, "name": "mon-staging", "private_ip": "10.0.1.3", "provider_override_args": ["vm.synced_folder './', '/vagrant', disabled: true, type: 'nfs'"]}, "module_stderr": "", "module_stdout": "--> Validating schema /home/kdas/code/securedrop/molecule/upgrade/molecule.yml.\nValidation completed successfully.\n", "msg": "MODULE FAILURE", "rc": 0}
    failed: [localhost] (item={'box': u'bento/ubuntu-14.04', 'provider_override_args': [u"vm.synced_folder './', '/vagrant', disabled: true, type: 'nfs'"], 'name': u'apt', 'groups': [u'aptservers'], 'memory': 256}) => {"changed": false, "item": {"box": "bento/ubuntu-14.04", "groups": ["aptservers"], "memory": 256, "name": "apt", "provider_override_args": ["vm.synced_folder './', '/vagrant', disabled: true, type: 'nfs'"]}, "msg": "ERROR: See log file '/tmp/molecule/securedrop/upgrade/vagrant-apt.err'"}
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=5    changed=0    unreachable=0    failed=1   

Also, I can only the upgrade_mon_staging vm running, but, no app vm.

@conorsch Please let me know if I am missing something before running make upgrade-start.

[conorsch]

@kushaldas The error message states "ERROR: See log file '/tmp/molecule/securedrop/upgrade/vagrant-app-staging.err'"}. What is the error reported in that file?

[kushaldas]

### 2019-01-11 16:11:22 ###
### 2019-01-11 16:11:22 ###
An error occurred while downloading the remote file. The error
message, if any, is reproduced below. Please fix this error and try
again.

OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
### 2019-01-11 16:43:42 ###
### 2019-01-11 16:43:42 ###
An error occurred while downloading the remote file. The error
message, if any, is reproduced below. Please fix this error and try
again.

OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
### 2019-01-11 17:53:47 ###
### 2019-01-11 17:53:47 ###
The box you're attempting to add has no available version that
matches the constraints you requested. Please double-check your
settings. Also verify that if you specified version constraints,
that the provider you wish to use is available for these constraints.

Box: fpf/securedrop-app
Address: file:///home/kdas/code/securedrop/molecule/vagrant_packager/box_files/app_metadata.json
Constraints: 0.11.0
Available versions: 0.7.0, 0.8.0, 0.9.1
### 2019-01-11 18:51:36 ###
### 2019-01-11 18:51:36 ###
The box you're attempting to add has no available version that
matches the constraints you requested. Please double-check your
settings. Also verify that if you specified version constraints,
that the provider you wish to use is available for these constraints.

Box: fpf/securedrop-app
Address: file:///home/kdas/code/securedrop/molecule/vagrant_packager/box_files/app_metadata.json
Constraints: 0.11.0
Available versions: 0.7.0, 0.8.0, 0.9.1
### 2019-01-11 18:54:08 ###
### 2019-01-11 18:54:08 ###
The box you're attempting to add has no available version that
matches the constraints you requested. Please double-check your
settings. Also verify that if you specified version constraints,
that the provider you wish to use is available for these constraints.

Box: fpf/securedrop-app
Address: file:///home/kdas/code/securedrop/molecule/vagrant_packager/box_files/app_metadata.json
Constraints: 0.11.0
Available versions: 0.7.0, 0.8.0, 0.9.1
### 2019-01-11 18:55:59 ###
### 2019-01-11 18:55:59 ###
The box you're attempting to add has no available version that
matches the constraints you requested. Please double-check your
settings. Also verify that if you specified version constraints,
that the provider you wish to use is available for these constraints.

Box: fpf/securedrop-app
Address: file:///home/kdas/code/securedrop/molecule/vagrant_packager/box_files/app_metadata.json
Constraints: 0.11.0
Available versions: 0.7.0, 0.8.0, 0.9.1

[conorsch]

What commit are you working from? The 0.11.0 metadata will only be present if you work from this branch. Post the sha sum of the commit you're on and I'll try to reproduce.

[kushaldas]

What commit are you working from? The 0.11.0 metadata will only be present if you work from this branch. Post the sha sum of the commit you're on and I'll try to reproduce.

commit cadd8786df40160b08624b00eac22bbdba120b78
Author: Conor Schaefer <conor@freedom.press>
Date:   Fri Jan 4 14:01:52 2019 -0800

@conorsch this is the latest commit.

[kushaldas]

Finally some new errors @conorsch please have a look

    TASK [Create molecule instance(s)] *********************************************
    changed: [localhost] => (item={'box': u'fpf/securedrop-app', 'name': u'app-staging', 'provider_override_args': [u"vm.synced_folder './', '/vagrant', disabled: true, type: 'nfs'"], 'box_url': u'../vagrant_packager/box_files/app_metadata.json', 'private_ip': u'10.0.1.2', 'groups': [u'securedrop_application_server', u'securedrop', u'staging'], 'memory': 1024, 'instance_raw_config_args': [u'ssh.insert_key = false']})
    failed: [localhost] (item={'box': u'fpf/securedrop-mon', 'name': u'mon-staging', 'provider_override_args': [u"vm.synced_folder './', '/vagrant', disabled: true, type: 'nfs'"], 'box_url': u'../vagrant_packager/box_files/mon_metadata.json', 'private_ip': u'10.0.1.3', 'groups': [u'securedrop_monitor_server', u'securedrop', u'staging'], 'memory': 1024, 'instance_raw_config_args': [u'ssh.insert_key = false']}) => {"changed": false, "item": {"box": "fpf/securedrop-mon", "box_url": "../vagrant_packager/box_files/mon_metadata.json", "groups": ["securedrop_monitor_server", "securedrop", "staging"], "instance_raw_config_args": ["ssh.insert_key = false"], "memory": 1024, "name": "mon-staging", "private_ip": "10.0.1.3", "provider_override_args": ["vm.synced_folder './', '/vagrant', disabled: true, type: 'nfs'"]}, "module_stderr": "", "module_stdout": "--> Validating schema /home/kdas/code/securedrop/molecule/upgrade/molecule.yml.\nValidation completed successfully.\n", "msg": "MODULE FAILURE", "rc": 0}
    failed: [localhost] (item={'box': u'bento/ubuntu-14.04', 'provider_override_args': [u"vm.synced_folder './', '/vagrant', disabled: true, type: 'nfs'"], 'name': u'apt', 'groups': [u'aptservers'], 'memory': 256}) => {"changed": false, "item": {"box": "bento/ubuntu-14.04", "groups": ["aptservers"], "memory": 256, "name": "apt", "provider_override_args": ["vm.synced_folder './', '/vagrant', disabled: true, type: 'nfs'"]}, "module_stderr": "", "module_stdout": "--> Validating schema /home/kdas/code/securedrop/molecule/upgrade/molecule.yml.\nValidation completed successfully.\n", "msg": "MODULE FAILURE", "rc": 0}
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=5    changed=0    unreachable=0    failed=1   

I can see upgrade_app_staging and upgrade_mon_staging and upgrade_apt running.

more update:

The upgrade_apt vm does not have any 10.0.1.x series IP as it has only one NIC, maybe that is the problem. It seems that is not there in the molecule anyway.

[kushaldas]

Finally things are working for me after @emkll told me the magic line of destroying the scenarios. The PR looks good. Though, I would love to see a future PR (or a new commit here) explaining how to destroy any borked instance.

Tried both test-local and test-qa scenarios.

[conorsch]

Though, I would love to see a future PR (or a new commit here) explaining how to destroy any borked instance.

Good flag, @kushaldas. I'm going to merge this, please feel free to follow-up with a clarification fix adding the changes you propose!