Skip to content

0.5.0 Test Plan

Jump to bottom
Kevin O'Gorman edited this page Nov 4, 2020 · 16 revisions

This test plan covers release 0.5.0 of the SecureDrop Workstation, which will ship consolidated templates via the preflight updater (#471), as well as other unreleased changes from main.

In addition to an RPM update, we will simultaneously issue new releases of most SecureDrop Workstation components, including the SecureDrop Client. These have their own versioning, but for simplicity, this issue uses "0.5.0" as a shorthand for the cross-component release.

Before beginning to test, please claim a column in the 0.5.0 test matrix and fill out the first 3 fields.

To set up the environment for testing correctly, you must work through the Template Consolidation scenario first, choosing one of the 3 install types:

  • Fresh install (self explanatory)
  • Happy Path - assumes the user follows the upgrade instructions for 0.5.0
  • Sad Path - assumes the user does not follow the upgrade instructions for 0.5.0

If time allows, standard acceptance tests should also be completed for each release candidate.

Then complete the rest of the test scenarios (or as many as possible).

Once finished, please append your completed test plan to the release issue and update the test matrix with your results for each tested scenario.

Release-specific test plan

Test parameters

  • Hardware target: e.g. Thinkpad X1 20KH002JUS
  • Install type: fresh install/"happy path" upgrade/"sad path" upgrade
  • securedrop-workstation-dom0-config version: Check via dnf list in dom0
  • Release candidate version: 0.5.0-rcN

Scenario: Template consolidation:

To prepare the test environment, complete the Server setup section below. Then complete the section for your choice of QA scenarios.

Server setup:

  • Ensure that your server is populated with at least 10 sources and with some file submissions.
  • Create a user via the Journalist Interface and record their login credentials, referenced as account X. Log in as account X via the Journalist Interface, and reply to some sources.

Install Type 1: Fresh Install:

  • If an existing production install is being used for testing, remove it completely with the following commands in dom0:

    securedrop-admin --uninstall
qvm-kill securedrop-workstation-buster
sudo dnf remove qubes-template-securedrop-workstation-buster
sudo dnf clean all

  • Follow the installation instructions to copy the submission key and journalist interface config details into place, then follow the steps below to download and install the workstation RPM.

  • in a networked VM (work, say):

    • download the latest securedrop-workstation-dom0-config version (sdw-latest.rpm, say) from https://yum-test.securedrop.org/workstation/dom0/f25/
    • clone the https://github.com/freedomofpress/securedrop-workstation.git repo and ensure that it is on the main branch

  • in dom0:

    • transfer the RPM to dom0 using qvm-run --pass-io work "cat sdw-latest.rpm" > sdw-latest.rpm
    • transfer the repo to dom0 using qvm-run --pass-io work "tar -c -C /home/user securedrop-workstation" | tar xvf -
    • install the RPM using sudo dnf install sdw-latest.rpm
    • update the Salt config to use QA repos using sudo bash ~/securedrop-workstation/utils/qa-switch.sh
    • set up the workstation using the command sdw-admin --apply

  • Verify that the installation completes successfully.

  • In dom0, run qvm-ls --tags sd-workstation, and verify that:

    • the only TemplateVMs listed are sd-(large|small)-buster-template) and securedrop-workstation-buster.
    • sd-app, sd-log, sd-proxy, and sd-gpg are based on sd-small-buster-template
    • sd-devices-dvm and sd-viewer are based on sd-large-buster-template

Install Type 2: Happy Path Upgrade:

  • Complete a production workstation setup following the installation instructions.
  • Start the SecureDrop Client and Verify that its version in the login dialog footer is 0.2.1
  • As the primary test journalist user, submit a reply to a source via the client.
  • in a networked VM (work, say):
    • clone the https://github.com/freedomofpress/securedrop-workstation.git repo and ensure that it is on the main branch
  • in dom0:
    • transfer the repo to dom0 using qvm-run --pass-io work "tar -c -C /home/user securedrop-workstation" | tar xvf -
    • update the Salt config to use QA repos using sudo bash ~/securedrop-workstation/utils/qa-switch.sh
    • update the workstation config RPM via sudo qubes-dom0-update -y and verify that it is now the release candidate with dnf list securedrop-workstation-dom0-config
    • verify that the pre-consolidation templates are still in use via qvm-ls --tags sd-workstation
    • update the Salt config again, using sudo bash ~/securedrop-workstation/utils/qa-switch.sh
    • remove the updater flag files, if they exist, with rm ~/.securedrop_launcher/sdw-{last_updated,update-status}
  • Double-click the SecureDrop icon to start the updater, and run tail -f ~/securedrop_launcher/logs/launcher.log in dom0
  • Verify that the installation completes successfully.
  • In dom0, run `qvm-ls --tags sd-workstation, and verify that:
    • the only TemplateVMs listed for AppVMs are sd-(large|small)-buster-template) and securedrop-workstation-buster.
    • sd-app, sd-log, sd-proxy, and sd-gpg are based on sd-small-buster-template
    • sd-devices and sd_viewer are based on sd-large-buster-template
  • Perform a timed run of the updater via the dom0 command: time /opt/securedrop/launcher/sdw-launcher.py --skip-delta=0 and record the time here: e.g. 12 minutes 32 seconds

Install Type 3: Sad Path Upgrade:

  • Complete a production workstation setup following the installation instructions.
  • Start the SecureDrop Client and Verify that its version in the login dialog footer is 0.2.1
  • As the primary test journalist user, submit a reply to a source via the client.
  • in a networked VM (work, say):
    • clone the https://github.com/freedomofpress/securedrop-workstation.git repo and ensure that it is on the main branch
  • in dom0:
    • transfer the repo to dom0 using qvm-run --pass-io work "tar -c -C /home/user securedrop-workstation" | tar xvf -
    • update the Salt config to use QA repos using sudo bash ~/securedrop-workstation/utils/qa-switch.sh
    • remove the updater flag files, if they exist, with rm ~/.securedrop_launcher/sdw-{last_updated,update-status}
  • Reboot the machine. When the GUI updater launches, leave it open without clicking Continue
  • in dom0:
    • verify the config RPM version is 0.4.0.1-fc25, with dnf list securedrop-workstation-dom0-config
    • verify that the pre-consolidation templates are still in use via qvm-ls --tags sd-workstation
  • In the updater, click Continue to start the update, and run tail -f ~/securedrop_launcher/logs/launcher.log in dom0
  • Verify that the updater completes without error
  • Verify that the client fails to start when the Securedrop icon is double-clicked.
  • in dom0:
    • verify the config RPM version is now 0.5.0.*-fc25, with dnf list securedrop-workstation-dom0-config
    • update the Salt config to use QA repos using sudo bash ~/securedrop-workstation/utils/qa-switch.sh
    • run sdw-admin --apply to apply the correct template configuration.
  • Verify that the installation completes successfully.
  • Verify that the client starts when the SecureDrop desktop icon is double-clicked, and its version is 0.2.1-dev-20201029-164632
  • In dom0, run qvm-ls --tags sd-workstation, and verify that:
    • the only TemplateVMs listed are sd-(large|small)-buster-template) and securedrop-workstation-buster.
    • sd-app, sd-log, sd-proxy, and sd-gpg are based on sd-small-buster-template
    • sd-devices and sd-viewer are based on sd-large-buster-template
  • Verify that client login and basic functionality are working.
  • Perform a timed run of the updater via the dom0 command: time /opt/securedrop/launcher/sdw-launcher.py --skip-delta=0 and record the time here: e.g. 12 minutes 32 seconds

Post-install: Reset Logging

  • on sd-log:

    • archive the logging directory with:

      tar -zcvf log_archive.tar.gz ~/QubesIncomingLogs
rm -rf ~/QubesIncomingLogs/*
  • on dom0:
    • restart sd_log via qvm-kill sd-log && qvm-start sd-log

Scenario: Acceptance Tests

Scenario: Use of sdw-admin

Test case: securedrop-admin to sdw-admin rename (Issue: #586; PR: #596)

  • Observe that securedrop-admin can no longer be run in dom0
  • Observe that you can now access the tool with sdw-admin in dom0
  • Observe that sdw-admin --validate still functions as expected

Scenario: Running the preflight updater

Test case: Text scaling in preflight updater (Issue: #597; PR: #599)

  • Under "System ▶ Appearance ▶ Fonts" in dom0, note the current font size of the "Default Font", then increase it to a very high value, e.g., 17.
  • Run the GUI updater, via /opt/securedrop/launcher/sdw-launcher.py --skip-delta 0. Complete the updater process while observing the following:
  • Observe that no text in the updater GUI is cut off
  • Observe that you cannot cause text to be cut off by resizing the window
  • After the update completes, restore the font size to the previous value.

Scenario: Using the SecureDrop Client

Test case: Preview snippets (Issue: https://github.com/freedomofpress/securedrop-client/issues/1121; PR: https://github.com/freedomofpress/securedrop-client/pull/1131)

  • Log into the SecureDrop Client
  • Before the client has finished decrypting all messages and replies, send a reply to an existing user
  • Observe that a preview of that reply is shown underneath the source name in the source list to the left, once it has been successfully sent
  • Wait for all messages/replies to be decrypted
  • Observe that during the sync, the preview snippet continues to show the reply you sent

Test case: Window resizing (Issues: https://github.com/freedomofpress/securedrop-client/issues/1109 and https://github.com/freedomofpress/securedrop-client/issues/1120; PRs: https://github.com/freedomofpress/securedrop-client/pull/1130 and https://github.com/freedomofpress/securedrop-client/pull/1145)

  • Observe that the client window is maximized with no part of the client hidden from view
  • Click on the square resize icon in the upper right-hand corner
  • Observe that the client window is changed to a smaller size, with no part of the window hidden from view (horizontal scrolling within the conversation view may be required)
  • Resize the client window manually by dragging its corner
  • Observe for multiple sources that during resizing, source abbreviations are elided at lower resolutions as shown in this GIF
  • Click a few times on the square resize icon in the upper right corner
  • Observe that the client switches back and forth between maximum size and your chosen size, with no part of the window hidden from view
  • If a second monitor is available, move the client window to a second monitor and repeat the previous step
  • Observe that the client switches back and forth between maximum size and your chosen size, with no part of the window hidden from view

Test case: Reply badges feature (Issue: https://github.com/freedomofpress/securedrop-client/issues/76; PR: https://github.com/freedomofpress/securedrop-client/pull/1142)

  • Log into the SecureDrop Client and wait for it to sync.
  • Observe that all previously sent replies show a two-letter badge representing the reply author. If first name and last name are set, lower-case initials will be used; otherwise, the first two letters of the username will be used.
  • Send a reply.
  • Observe that the sent reply shows a two-letter badge representing your account.
  • Observe that the badge for replies you have sent is in a different color than the badge for replies sent by other users.
  • Through the Journalist Interface, delete account X. (Stay logged into the JI for a later step.)
  • Wait for the SecureDrop Client to synchronize with the server.
  • Observe that replies sent by account X are now attributed to a placeholder user, represented by a star cluster icon.
  • Through the Journalist Interface, change the first and last name of the user logged into the SecureDrop Client to names with different initial.
  • Wait for the SecureDrop Client to synchronize with the server.
  • Observe that the updated initials are shown in reply badges, and in the login badge (top left corner).
  • Switch to offline mode
  • Observe that all reply badges are still present, but are now in a uniform color, even if sent by the previously logged in user.

Test case: Decryption error styling (Issue: https://github.com/freedomofpress/securedrop-client/issues/1150; PR: https://github.com/freedomofpress/securedrop-client/pull/1151)

  • Log out of the SecureDrop Client
  • Via the Source Interface, send a message as a new source
  • Via the Journalist Interface, send a reply to that source
  • In sd-gpg, temporarily move ~/.gnupg to ~/.gnupg.bak to cause decryption errors
  • Log into the SecureDrop Client and wait for it to sync
  • Observe that message and reply are styled identically, in gray and italics: "cannot decrypt message" / "cannot decrypt reply"
  • In sd-gpg, move ~/.gnupg.bak back to ~/.gnupg

Scenario: Investigating logs

Test case: Sorting logs by hostname (Issue: #583; PR: https://github.com/freedomofpress/securedrop-log/pull/18)

  • Observe that no host directory exists in ~/QubesIncomingLogs on sd-log (only relevant for "Fresh install" scenario, otherwise the directory will be present)
  • Run the command logger LOGME in sd-app, sd-whonix, and sd-log.
  • Observe that the line was added to ~/QubesIncomingLogs/sd-app and ~/QubesIncomingLogs/sd-whonix
  • Observe that no sd-log directory exists in ~/QubesIncomingLogs on sd-log.
Clone this wiki locally