Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[0.5.0] Several mimetype-related tests are failing #634

Closed
conorsch opened this issue Nov 4, 2020 · 7 comments
Closed

[0.5.0] Several mimetype-related tests are failing #634

conorsch opened this issue Nov 4, 2020 · 7 comments
Projects
SecureDrop Team Board

Comments

@conorsch
Copy link
Contributor

conorsch commented Nov 4, 2020

Last tested on PR #632 (which will likely become 0.5.0-rc3).

failing tests output 
======================================================================
ERROR: test_open_in_dvm_desktop (test_sd_devices.SD_Devices_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/securedrop-workstation/tests/test_sd_devices.py", line 40, in test_open_in_dvm_desktop
    contents = self._get_file_contents("/usr/share/applications/open-in-dvm.desktop")
  File "/home/user/securedrop-workstation/tests/base.py", line 62, in _get_file_contents
    contents = subprocess.check_output(cmd).decode("utf-8")
  File "/usr/lib64/python3.5/subprocess.py", line 316, in check_output
    **kwargs).stdout
  File "/usr/lib64/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['qvm-run', '-p', 'sd-devices-dvm', 'sudo /bin/cat /usr/share/applications/open-in-dvm.desktop']' returned non-zero exit status 1

======================================================================
FAIL: test_mime_types (test_sd_devices.SD_Devices_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/securedrop-workstation/tests/test_sd_devices.py", line 37, in test_mime_types
    self.assertEqual(actual_app, expected_app)
AssertionError: '' != 'open-in-dvm.desktop'
+ open-in-dvm.desktop

======================================================================
FAIL: test_mime_types (test_viewer.SD_Viewer_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/securedrop-workstation/tests/test_viewer.py", line 45, in test_mime_types
    self.assertEqual(actual_app, expected_app)
AssertionError: 'libreoffice-calc.desktop' != 'libreoffice-base.desktop'
- libreoffice-calc.desktop
?             ^ ^^
+ libreoffice-base.desktop
?             ^ ^^


======================================================================
FAIL: test_all_sd_vms_uptodate (test_vms_platform.SD_VM_Platform_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/securedrop-workstation/tests/test_vms_platform.py", line 169, in test_all_sd_vms_uptodate
    self._ensure_packages_up_to_date(vm)
  File "/home/user/securedrop-workstation/tests/test_vms_platform.py", line 95, in _ensure_packages_up_to_date
    self.assertEqual(results, "Listing...", fail_msg)
AssertionError: 'Listing...\nsecuredrop-workstation-config/unknown 0[43 chars]ter]' != 'Listing...'
- Listing...
?           -
+ Listing...- securedrop-workstation-config/unknown 0.1.5+buster all [upgradable from: 0.1.3+buster] : Unapplied updates for VM 'sd-log'

----------------------------------------------------------------------
Ran 79 tests in 354.716s

FAILED (failures=3, errors=1)
Makefile:109: recipe for target 'test' failed
make: *** [test] Error 1

Let's make sure to get these down to zero prior to releasing 0.5.0 final.
@conorsch conorsch added this to SecureDrop Sprint #62- 10/28-11/12 in SecureDrop Team Board Nov 4, 2020
@zenmonkeykstop
Copy link
Contributor

zenmonkeykstop commented Nov 4, 2020 •
edited

Output on a fresh make dev for #632 for me:

[kog@dom0 securedrop-workstation]$ make test
python3 -m unittest discover -v tests
test_gpg_domain_configured (test_app.SD_App_Tests) ... ok
test_logging_configured (test_app.SD_App_Tests) ... ok
test_mimeapps (test_app.SD_App_Tests) ... ok
test_mimeapps_functional (test_app.SD_App_Tests) ... ok
test_open_in_dvm_desktop (test_app.SD_App_Tests) ... ok
test_sd_client_apparmor (test_app.SD_App_Tests) ... ok
test_sd_client_config (test_app.SD_App_Tests) ... ok
test_sd_client_dependencies_installed (test_app.SD_App_Tests) ... ok
test_sd_client_package_installed (test_app.SD_App_Tests) ... ok
test_Templates_cleaned_up (test_dom0_config.SD_Qubes_Dom0_Templates_Tests) ... ok
test_vms_to_update_are_tagged (test_dom0_config.SD_Qubes_Dom0_Templates_Tests) ... ok
test_rpm_repo_config (test_dom0_rpm_repo.SD_Dom0_Rpm_Repo_Tests) ... ok
test_rpm_repo_public_key (test_dom0_rpm_repo.SD_Dom0_Rpm_Repo_Tests) ... ok
test_gpg_domain_configured (test_gpg.SD_GPG_Tests) ... ok
test_logging_disabled (test_gpg.SD_GPG_Tests) ... ok
test_sd_gpg_timeout (test_gpg.SD_GPG_Tests) ... ok
test_we_have_the_key (test_gpg.SD_GPG_Tests) ... ok
test_gpg_domain_configured (test_log_vm.SD_Log_Tests) ... ok
test_log_dirs_properly_named (test_log_vm.SD_Log_Tests) ... ok
test_log_utility_installed (test_log_vm.SD_Log_Tests) ... ok
test_logs_are_flowing (test_log_vm.SD_Log_Tests) ... ok
test_redis_service_running (test_log_vm.SD_Log_Tests) ... ok
test_sd_log_has_no_custom_rsyslog (test_log_vm.SD_Log_Tests) ... ok
test_sd_log_package_installed (test_log_vm.SD_Log_Tests) ... ok
test_sd_log_redis_is_installed (test_log_vm.SD_Log_Tests) ... ok
test_sd_log_service_running (test_log_vm.SD_Log_Tests) ... ok
test_do_not_open_here (test_proxy_vm.SD_Proxy_Tests) ... ok
test_gpg_domain_configured (test_proxy_vm.SD_Proxy_Tests) ... ok
test_logging_configured (test_proxy_vm.SD_Proxy_Tests) ... ok
test_mime_types (test_proxy_vm.SD_Proxy_Tests) ... ok
test_sd_proxy_package_installed (test_proxy_vm.SD_Proxy_Tests) ... ok
test_sd_proxy_rpc_spec (test_proxy_vm.SD_Proxy_Tests) ... ok
test_sd_proxy_writable_config_dir (test_proxy_vm.SD_Proxy_Tests) ... ok
test_sd_proxy_yaml_config (test_proxy_vm.SD_Proxy_Tests) ... ok
test_whonix_ws_repo_absent (test_proxy_vm.SD_Proxy_Tests) ... ok
test_Policies (test_qubes_rpc.SD_Qubes_Rpc_Tests) ... ok
test_current_fedora_for_sys_vms (test_qubes_vms.SD_Qubes_VM_Tests) ... ok
test_current_whonix_vms (test_qubes_vms.SD_Qubes_VM_Tests) ... ok
test_files_are_properly_copied (test_sd_devices.SD_Devices_Tests) ... ok
test_gpg_domain_configured (test_sd_devices.SD_Devices_Tests) ... ok
test_logging_configured (test_sd_devices.SD_Devices_Tests) ... ok
test_mime_types (test_sd_devices.SD_Devices_Tests) ... FAIL
test_open_in_dvm_desktop (test_sd_devices.SD_Devices_Tests) ... /bin/cat: /usr/share/applications/open-in-dvm.desktop: No such file or directory
ERROR
test_sd_export_package_installed (test_sd_devices.SD_Devices_Tests) ... ok
test_accept_sd_xfer_extracted_file (test_sd_whonix.SD_Whonix_Tests) ... ok
test_gpg_domain_configured (test_sd_whonix.SD_Whonix_Tests) ... ok
test_logging_configured (test_sd_whonix.SD_Whonix_Tests) ... ok
test_sd_whonix_repo_enabled (test_sd_whonix.SD_Whonix_Tests) ... ok
test_sd_whonix_verify_tor_config (test_sd_whonix.SD_Whonix_Tests) ... ok
test_v3_auth_private_file (test_sd_whonix.SD_Whonix_Tests) ... skipped 'Onion v3 address is not setup'
test_whonix_torrc (test_sd_whonix.SD_Whonix_Tests) ... ok
test_rpm_repo_public_key (test_sys_firewall.SD_Sys_Firewall_Tests) ... ok
test_rpm_repo_public_key_script (test_sys_firewall.SD_Sys_Firewall_Tests) ... ok
test_files_are_properly_copied (test_sys_usb.SD_SysUSB_Tests) ... ok
test_gpg_domain_configured (test_viewer.SD_Viewer_Tests) ... ok
test_logging_configured (test_viewer.SD_Viewer_Tests) ... ok
test_mime_types (test_viewer.SD_Viewer_Tests) ... FAIL
test_redis_packages_not_installed (test_viewer.SD_Viewer_Tests) ... ok
test_sd_viewer_evince_installed (test_viewer.SD_Viewer_Tests) ... ok
test_sd_viewer_libreoffice_installed (test_viewer.SD_Viewer_Tests) ... ok
test_sd_viewer_metapackage_installed (test_viewer.SD_Viewer_Tests) ... ok
test_expected (test_vms_exist.SD_VM_Tests) ... ok
test_sd_app_config (test_vms_exist.SD_VM_Tests) ... ok
test_sd_gpg_config (test_vms_exist.SD_VM_Tests) ... ok
test_sd_log_config (test_vms_exist.SD_VM_Tests) ... ok
test_sd_proxy_config (test_vms_exist.SD_VM_Tests) ... ok
test_sd_proxy_template (test_vms_exist.SD_VM_Tests) ... ok
test_sd_viewer_config (test_vms_exist.SD_VM_Tests) ... ok
test_sd_whonix_config (test_vms_exist.SD_VM_Tests) ... ok
test_sd_workstation_template (test_vms_exist.SD_VM_Tests) ... ok
test_all_fedora_vms_uptodate (test_vms_platform.SD_VM_Platform_Tests) ... ok
test_all_jessie_backports_disabled (test_vms_platform.SD_VM_Platform_Tests) ... ok
test_all_sd_vm_apt_sources (test_vms_platform.SD_VM_Platform_Tests) ... ok
test_all_sd_vm_platforms (test_vms_platform.SD_VM_Platform_Tests) ... ok
test_all_sd_vms_uptodate (test_vms_platform.SD_VM_Platform_Tests) ... FAIL
test_debian_keyring_config (test_vms_platform.SD_VM_Platform_Tests) ... ok
test_dispvm_default_platform (test_vms_platform.SD_VM_Platform_Tests) ... ok
test_sd_proxy_template (test_vms_platform.SD_VM_Platform_Tests) ... ok
test_sys_vms_use_supported_fedora (test_vms_platform.SD_VM_Platform_Tests) ... ok

======================================================================
ERROR: test_open_in_dvm_desktop (test_sd_devices.SD_Devices_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kog/securedrop-workstation/tests/test_sd_devices.py", line 40, in test_open_in_dvm_desktop
    contents = self._get_file_contents("/usr/share/applications/open-in-dvm.desktop")
  File "/home/kog/securedrop-workstation/tests/base.py", line 62, in _get_file_contents
    contents = subprocess.check_output(cmd).decode("utf-8")
  File "/usr/lib64/python3.5/subprocess.py", line 316, in check_output
    **kwargs).stdout
  File "/usr/lib64/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['qvm-run', '-p', 'sd-devices-dvm', 'sudo /bin/cat /usr/share/applications/open-in-dvm.desktop']' returned non-zero exit status 1

======================================================================
FAIL: test_mime_types (test_sd_devices.SD_Devices_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kog/securedrop-workstation/tests/test_sd_devices.py", line 37, in test_mime_types
    self.assertEqual(actual_app, expected_app)
AssertionError: '' != 'open-in-dvm.desktop'
+ open-in-dvm.desktop

======================================================================
FAIL: test_mime_types (test_viewer.SD_Viewer_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kog/securedrop-workstation/tests/test_viewer.py", line 45, in test_mime_types
    self.assertEqual(actual_app, expected_app)
AssertionError: 'libreoffice-calc.desktop' != 'libreoffice-base.desktop'
- libreoffice-calc.desktop
?             ^ ^^
+ libreoffice-base.desktop
?             ^ ^^


======================================================================
FAIL: test_all_sd_vms_uptodate (test_vms_platform.SD_VM_Platform_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kog/securedrop-workstation/tests/test_vms_platform.py", line 169, in test_all_sd_vms_uptodate
    self._ensure_packages_up_to_date(vm)
  File "/home/kog/securedrop-workstation/tests/test_vms_platform.py", line 95, in _ensure_packages_up_to_date
    self.assertEqual(results, "Listing...", fail_msg)
AssertionError: 'Listing...\nsecuredrop-workstation-config/unknown 0[43 chars]ter]' != 'Listing...'
- Listing...
?           -
+ Listing...- securedrop-workstation-config/unknown 0.1.5+buster all [upgradable from: 0.1.3+buster] : Unapplied updates for VM 'sd-log'

----------------------------------------------------------------------
Ran 79 tests in 162.291s
  • The error and first failure related to open-in-dvm.desktop are probably a result of consolidation changes - investigation is required to see if the tests need changing or if the config isn't as expected.
  • The failure related to libreofffice-base.deskstop looks like a result of bad test data and updates to libreoffice itself, those mime types are now associated with the specific application desktop file that handles them as opposed to the base one.
  • The last failure looks to be a result of using nightlies and is probably safe to ignore

@zenmonkeykstop zenmonkeykstop mentioned this issue Nov 4, 2020
Merged
9 tasks
@emkll
Copy link
Contributor

emkll commented Nov 4, 2020 •
edited

I have also observed these errors locally, however after running the updater (unclear and unlikely that had an impact) and shutting down all the workstation VMs, most tests were once again passing on the branch in 632.

It seems like the issue is the DispVM templates which weren't shutdown after the initial provisioning run: sd-devices-dvm and sd-viewer were out of sync with the template (they were using config 1.3 which is the one "baked into" the base template, which means securedrop-workstation-buster was not shutdown after updating and before it is clone to the small and large templates.

We should check the shutdown order within the GUI updater, and since a normal user would have to run through an updater cycle to get their workstation in order after running the manual --apply command, this could address the situation in non-dev scenarios, but would obvious be an issue for dev

@conorsch
Copy link
Contributor Author

conorsch commented Nov 4, 2020

Thanks @emkll, that's helpful. A full host reboot likely would have straightened me out, I'll confirm that today.

@conorsch
Copy link
Contributor Author

conorsch commented Nov 4, 2020

Encountered again on 0.5.0-rc3 as part of QA for fresh install. Ran the updater, then manually restarted the VMs, and all tests were passing. Running make-dev and make-test from the same code, via the main branch, showed all tests passing immediately, so I suspect the state drift is related to our use of the qa-switch.sh util mandated by the test plan. Removing the "release-blocker" label for 0.5.0, but leaving open, so we can observe behavior of the prod packages and decide on whether a 0.5.1 is warranted.

@conorsch conorsch mentioned this issue Nov 5, 2020
Closed
36 tasks
@emkll
Copy link
Contributor

emkll commented Nov 5, 2020

Related to freedomofpress/securedrop-updater#35

@emkll
Copy link
Contributor

emkll commented Nov 10, 2020

Closing, it seems like freedomofpress/securedrop-updater#35 is the underlying issue here, and 0.5.0 has been released.

@emkll emkll closed this as completed Nov 10, 2020
SecureDrop Team Board automation moved this from SecureDrop Sprint #62- 10/28-11/12 to Done Nov 10, 2020
@conorsch
Copy link
Contributor Author

FWIW I'm no longer able to reproduce these test failures. On latest main. i.e. 6b4896d, I'm seeing 100% of tests passing. freedomofpress/securedrop-updater#35 should still be addressed, but at least our tests are passing again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
No open projects
SecureDrop Team Board
Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@conorsch @zenmonkeykstop @emkll