Investigate flaky tests #1139

sssoleileraaa · 2020-08-12T19:17:52Z

Description

I've seen the following tests in the client behave flakily :):

test_PrintDialog__show_insert_usb_message (see circleci)

    def stop_animate_header(self):
>       self.header_icon.setVisible(True)
E       RuntimeError: wrapped C/C++ object of type SvgLabel has been deleted

test_ExportDialog__export_file (see circleci)

        # Connect parent signals to slots
>       self.continue_button.setEnabled(False)
E       RuntimeError: wrapped C/C++ object of type QPushButton has been deleted

test_PrintDialog__print_file (see circleci)

        # Connect parent signals to slots
>       self.continue_button.setEnabled(False)
E       RuntimeError: wrapped C/C++ object of type QPushButton has been deleted

test_ModalDialog_animation_of_activestate (see circle)

>       self.error_details.setStyleSheet("")
E       RuntimeError: wrapped C/C++ object of type QLabel has been deleted

Further investigation needs to be done around why the dialog tests appear to be flaky - could have to do with needing make test assertions before signal handlers are initiated, e.g. the close signal handler will close the dialog so making an assertion on the dialog after it has been closed could cause an issue.

STR

See #1139 (comment)

The text was updated successfully, but these errors were encountered:

sssoleileraaa · 2020-08-12T19:48:30Z

This could be related to #1134 because that's when I started seeing the flakes. Keeping track of how often this starts happening here.

Update

We've confirmed that the problem persists even when we revert #1134

sssoleileraaa · 2020-08-12T19:50:26Z

Also the flakes only seem to occur in circleci. You can see how rerunning the workflow on the main branch resolved it here for #347: https://app.circleci.com/pipelines/github/freedomofpress/securedrop-client?branch=main.

Update

We're able to reproduce locally now (see STR in this Issue's description)

eloquence · 2020-08-21T00:08:28Z

We'll continue to keep an eye on this. @rmol has encountered these flakes as well, and has committed to at least a brief investigation during the 8/20-9/2 sprint.

eloquence · 2020-09-03T19:01:28Z

No progress on this last sprint but test integrity is important, so @rmol will take a look 9/3-9/17, same timebox.

eloquence · 2020-09-17T21:47:34Z

We'll continue to watch for flaky tests on client PRs during the 9/17-10/1 sprint and @rmol will help investigate as warranted.

emkll · 2020-10-01T18:13:47Z

I observed a failed test run in https://app.circleci.com/pipelines/github/freedomofpress/securedrop-client/395/workflows/53167775-c4c5-4d9e-abc0-51a50d1180ab/jobs/4685

sssoleileraaa · 2020-10-28T18:46:32Z

new example of the same issue still occurring: https://circleci.com/api/v1.1/project/github/freedomofpress/securedrop-client/4822/output/103/0?file=true&allocation-id=5f99bacd4771a47ad89e8a53-0-build%2F5A1221DF

eloquence · 2020-10-28T23:04:16Z

Given these continued test flakes, we're keeping this on the 10/28-11/12 sprint for continued investigation. Would be good to better understand cause of the flake before the November 4 release, to reduce risk of unintended regression beyond the test flakiness.

sssoleileraaa · 2020-11-05T22:25:47Z

The nice thing about these flakes is that the second time I run ci, the tests seem to always pass.

eloquence · 2020-11-12T19:31:07Z

@creviera has committed to poking at John's findings in #1158 during the 11/12-11/25 sprint to see if we can get closer to a root cause here.

kushaldas · 2021-03-03T16:23:05Z

I also noticed this today in the CI for many times while reviewing #1222.
It happened on various different widgets.

https://app.circleci.com/pipelines/github/freedomofpress/securedrop-client/567/workflows/6823f779-39fd-4569-9d3d-cb03783941d3/jobs/5091
https://app.circleci.com/pipelines/github/freedomofpress/securedrop-client/567/workflows/6823f779-39fd-4569-9d3d-cb03783941d3/jobs/5092
https://app.circleci.com/pipelines/github/freedomofpress/securedrop-client/567/workflows/6823f779-39fd-4569-9d3d-cb03783941d3/jobs/5093
https://app.circleci.com/pipelines/github/freedomofpress/securedrop-client/567/workflows/6823f779-39fd-4569-9d3d-cb03783941d3/jobs/5094

emkll · 2021-03-26T15:43:50Z

I have observed three failures in a row on main this morning, all different tests:

tests/gui/test_widgets.py::test_ExportDialog__on_preflight_success: https://app.circleci.com/pipelines/github/freedomofpress/securedrop-client/608/workflows/c8010cbe-3f9d-417f-acb8-105d53746f36/jobs/5182
tests/gui/test_widgets.py::test_ExportDialog__show_starting_instructions: https://app.circleci.com/pipelines/github/freedomofpress/securedrop-client/608/workflows/ac89b43d-1d3c-4b59-8e58-dcff795ddb27/jobs/5180
tests/gui/test_widgets.py::test_PrintDialog__on_preflight_success_enabled_after_preflight_failure: https://app.circleci.com/pipelines/github/freedomofpress/securedrop-client/608/workflows/4b6fc463-45f7-44b8-b44c-2463a4cd2bf1/jobs/5178

sssoleileraaa · 2021-04-10T00:13:20Z

I had a little bit of time to look at this again and I think we need to look carefully at self.button_animation.frameChanged.connect(self.animate_activestate) and self.header_animation.frameChanged.connect(self.animate_header) to understand how the handlers might be called after the export/print dialogs are closed.

sssoleileraaa · 2021-04-10T00:15:20Z

As a reminder, we show an animation next to the file name in the header when we are booting up sd-devices and we show an animation in the continue_button when it's making a request to sd-devices

sssoleileraaa · 2021-04-13T20:48:45Z

I checked if removing the animations fixed the problem, and even though it made the test failures a lot less likely to occur (because most of the failures are when we access modal widgets to display our animations in) it did not fix the issue. We can still see flaky tests when they initialize a new export or print dialog (see https://app.circleci.com/pipelines/github/freedomofpress/securedrop-client/646/workflows/8237e0d1-9660-4d39-ad96-b2ba38aa2d6b/jobs/5294), which makes it seem more likely to be an issue with test concurrency + ordering since we run our tests in parallel and in random order. Testing methods such as _print_file will close a dialog that a test running in parallel might have just unknowningly sent a signal to. I think the next step is to confirm my hypothesis. Hopefully this is it and we just need to add a dialog fixture or something that provides dialog initialization and cleanup between tests.

sssoleileraaa · 2021-04-13T20:54:49Z

Oh, also, as a side note, https://circleci.com/api/v1.1/project/github/freedomofpress/securedrop-client/5298/output/103/0?file=true&allocation-id=6075fe02363fc6688fd0c788-0-build%2F51B3148A happened twice in the past 30 mins, so marking all our functional tests flaky might not be the best solution in the long run, but one flaky issue at a time...

eloquence · 2021-04-15T22:11:41Z

For the 4/15 sprint, @creviera has committed to resolving one type of test flake (possibly the animation-related ones). We'll keep chipping away at this.

sssoleileraaa · 2021-04-22T02:29:38Z

For testing, we can use this script to repro and to see that it is fixed.

Update

Meant to say when it is fixed

conorsch · 2021-04-23T00:31:41Z

Immediately post-merge of #1248, I saw the following failure in CircleCI:

failure output

=================================== FAILURES ===================================
______ test_ExportDialog__update_dialog_when_status_is_USB_NOT_CONNECTED _______

mocker = <pytest_mock.MockFixture object at 0x7fd114d40290>

    def test_ExportDialog__update_dialog_when_status_is_USB_NOT_CONNECTED(mocker):
>       dialog = ExportDialog(mocker.MagicMock(), "mock_uuid", "mock_filename")

tests/gui/test_widgets.py:3827: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <securedrop_client.gui.widgets.ExportDialog object at 0x7fd114eb6690>
controller = <MagicMock id='140535974330704'>, file_uuid = 'mock_uuid'
file_name = 'mock_filename'

    def __init__(self, controller: Controller, file_uuid: str, file_name: str):
        super().__init__()
    
        self.controller = controller
        self.file_uuid = file_uuid
        self.file_name = SecureQLabel(
            file_name, wordwrap=False, max_length=self.FILENAME_WIDTH_PX
        ).text()
        self.error_status = ""  # Hold onto the error status we receive from the Export VM
    
        # Connect controller signals to slots
        self.controller.export.preflight_check_call_success.connect(self._on_preflight_success)
        self.controller.export.preflight_check_call_failure.connect(self._on_preflight_failure)
        self.controller.export.export_usb_call_success.connect(self._on_export_success)
        self.controller.export.export_usb_call_failure.connect(self._on_export_failure)
    
        # Connect parent signals to slots
>       self.continue_button.setEnabled(False)
E       RuntimeError: wrapped C/C++ object of type QPushButton has been deleted

securedrop_client/gui/widgets.py:2730: RuntimeError

sssoleileraaa · 2021-04-23T00:36:54Z

yup! #1248 fixed yet another flaky test. and just a heads up that we also have flaky functional tests. but what you linked to here is the runtime error mentioned in this issue which I'm trying to fix or minimize in #1243

sssoleileraaa · 2021-11-04T16:24:15Z

Seeing this issue again: https://app.circleci.com/pipelines/github/freedomofpress/securedrop-client/1139/workflows/d26456f0-7e44-41ef-9d0a-9ad89b47c956/jobs/6370

Looks like another issue between starting/stopping animations and dialogs.

Also: https://app.circleci.com/pipelines/github/freedomofpress/securedrop-client/1139/workflows/d26456f0-7e44-41ef-9d0a-9ad89b47c956/jobs/6370

sssoleileraaa · 2021-12-22T02:15:23Z

bumping... we have more eyes on the project so others see this issue now too

sssoleileraaa · 2021-12-22T02:18:26Z

it's still pretty rare - nothing like it was before - but it's such a headache that i plan to take a peek 👀 again after the holiday

gonzalo-bulnes · 2021-12-22T02:19:41Z

@conorsch, @rocodes and I encountered this one today a few times.

Hypothesis:

Fact (I think!): Python references to Qt objects should prevent garbage collection of the underlying C++ object.
Idea: what if a Python reference was removed while an event is waiting to be processed by the Qt event loop?

@rocodes was noting that specially in a testing context we create and tear down objects very quickly.

Note: even if this hypothesis was true, I don't think that should be happening!! 😬

gonzalo-bulnes · 2021-12-22T02:21:22Z

I had a little bit of time to look at this again and I think we need to look carefully at self.button_animation.frameChanged.connect(self.animate_activestate) and self.header_animation.frameChanged.connect(self.animate_header) to understand how the handlers might be called after the export/print dialogs are closed.

With the above hypothesis in mind, it would make sense to me that the issue would occur particularly around signals.

sssoleileraaa · 2022-04-19T22:31:04Z

A new test failure occurred in https://app.circleci.com/pipelines/github/freedomofpress/securedrop-client/1955/workflows/b8a2a85f-6edb-44d9-b07b-bd65d35ddcf5/jobs/8077 and looks related to

securedrop-client/securedrop_client/db.py

Line 297 in 9ed27b4

if self.seen_messages.count() or self.is_read:

, but will have to investigate later:

tests/functional/test_seen.py::test_seen_and_unseen Fatal Python error: Aborted

The test usually passes so this might point us to a race.

emkll added this to SecureDrop Sprint #56 - 8/6-8/20 in SecureDrop Team Board Aug 13, 2020

eloquence removed this from SecureDrop Sprint #61 - 10/15-10/28 in SecureDrop Team Board Oct 15, 2020

eloquence added this to Maintenance period (Kanban mode) in SecureDrop Team Board Mar 16, 2021

eloquence moved this from 1.8.1 Release Period (Kanban Mode) to In Development in SecureDrop Team Board Apr 12, 2021

sssoleileraaa mentioned this issue Apr 13, 2021

debug flaky #1243

Closed

sssoleileraaa mentioned this issue Apr 23, 2021

use dialog fixtures and fix flaky tests #1249

Merged

3 tasks

emkll closed this as completed in #1249 Apr 27, 2021

SecureDrop Team Board automation moved this from In Development to Done Apr 27, 2021

sssoleileraaa reopened this Nov 4, 2021

This was referenced Apr 18, 2022

Update changelog for securedrop-client0 0.7.0-rc2 freedomofpress/securedrop-builder#303

Merged

Add changelog entry and update version to 0.7.0-rc2 #1479

Merged

sssoleileraaa moved this from Done to Near Term - SD Workstation in SecureDrop Team Board Apr 20, 2022

sssoleileraaa removed this from Near Term - SD Workstation in SecureDrop Team Board Apr 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Investigate flaky tests #1139

Investigate flaky tests #1139

sssoleileraaa commented Aug 12, 2020 •

edited

sssoleileraaa commented Aug 12, 2020 •

edited

sssoleileraaa commented Aug 12, 2020 •

edited

eloquence commented Aug 21, 2020

eloquence commented Sep 3, 2020

eloquence commented Sep 17, 2020

emkll commented Oct 1, 2020

sssoleileraaa commented Oct 28, 2020

eloquence commented Oct 28, 2020

sssoleileraaa commented Nov 5, 2020

eloquence commented Nov 12, 2020

kushaldas commented Mar 3, 2021 •

edited

emkll commented Mar 26, 2021 •

edited

sssoleileraaa commented Apr 10, 2021

sssoleileraaa commented Apr 10, 2021

sssoleileraaa commented Apr 13, 2021

sssoleileraaa commented Apr 13, 2021

eloquence commented Apr 15, 2021

sssoleileraaa commented Apr 22, 2021 •

edited

conorsch commented Apr 23, 2021

sssoleileraaa commented Apr 23, 2021

sssoleileraaa commented Nov 4, 2021 •

edited

sssoleileraaa commented Dec 22, 2021

sssoleileraaa commented Dec 22, 2021 •

edited

gonzalo-bulnes commented Dec 22, 2021

gonzalo-bulnes commented Dec 22, 2021

sssoleileraaa commented Apr 19, 2022

Investigate flaky tests #1139

Investigate flaky tests #1139

Comments

sssoleileraaa commented Aug 12, 2020 • edited

Description

STR

sssoleileraaa commented Aug 12, 2020 • edited

Update

sssoleileraaa commented Aug 12, 2020 • edited

Update

eloquence commented Aug 21, 2020

eloquence commented Sep 3, 2020

eloquence commented Sep 17, 2020

emkll commented Oct 1, 2020

sssoleileraaa commented Oct 28, 2020

eloquence commented Oct 28, 2020

sssoleileraaa commented Nov 5, 2020

eloquence commented Nov 12, 2020

kushaldas commented Mar 3, 2021 • edited

emkll commented Mar 26, 2021 • edited

sssoleileraaa commented Apr 10, 2021

sssoleileraaa commented Apr 10, 2021

sssoleileraaa commented Apr 13, 2021

sssoleileraaa commented Apr 13, 2021

eloquence commented Apr 15, 2021

sssoleileraaa commented Apr 22, 2021 • edited

Update

conorsch commented Apr 23, 2021

sssoleileraaa commented Apr 23, 2021

sssoleileraaa commented Nov 4, 2021 • edited

sssoleileraaa commented Dec 22, 2021

sssoleileraaa commented Dec 22, 2021 • edited

gonzalo-bulnes commented Dec 22, 2021

gonzalo-bulnes commented Dec 22, 2021

sssoleileraaa commented Apr 19, 2022

sssoleileraaa commented Aug 12, 2020 •

edited

sssoleileraaa commented Aug 12, 2020 •

edited

sssoleileraaa commented Aug 12, 2020 •

edited

kushaldas commented Mar 3, 2021 •

edited

emkll commented Mar 26, 2021 •

edited

sssoleileraaa commented Apr 22, 2021 •

edited

sssoleileraaa commented Nov 4, 2021 •

edited

sssoleileraaa commented Dec 22, 2021 •

edited