DeleteAndQuitTest fails for fennecProduction build type

[pocmo]

DeleteAndQuitTest is failing for the fennecProduction build type:
https://tools.taskcluster.net/groups/NUDVpqIwShOoOy05eeyNBw

SUITE: org.mozilla.fenix.utils.DeleteAndQuitTest
  TEST: delete everything and quit
    [Robolectric] org.mozilla.fenix.utils.DeleteAndQuitTest.delete everything and quit: sdk=28; resources=LEGACY
    [Robolectric] NOTICE: legacy resources mode is deprecated; see http://robolectric.org/migrating/#migrating-to-40
    I/AssetManager: no overlays.list file found
    I/MonitoringInstr: Instrumentation started!
    I/MonitoringInstr: Setting context classloader to 'org.robolectric.internal.AndroidSandbox$SdkSandboxClassLoader@136627fc', Original: 'org.robolectric.internal.AndroidSandbox$SdkSandboxClassLoader@136627fc'
  FAILURE

java.lang.AssertionError: Verification failed: call 3 of 7: Engine(#331).clearData(eq(mozilla.components.concept.engine.Engine$BrowsingData@c0), null(), eq(lambda {}), eq(lambda {}))). No matching calls found.

Calls to same method:
1) Engine(#331).clearData(mozilla.components.concept.engine.Engine$BrowsingData@6, null, lambda {}, lambda {})
2) Engine(#331).clearData(mozilla.components.concept.engine.Engine$BrowsingData@21, null, lambda {}, lambda {})
3) Engine(#331).clearData(mozilla.components.concept.engine.Engine$BrowsingData@10, null, lambda {}, lambda {})

Stack traces:

┆Issue is synchronized with this Jira Task

[pocmo]

CC @ekager @NotWoods You both seem to have worked on that test.

[sblatz]

This one seems to be an intermittent failure. I've had it pass/fail for me on my taskcluster after re-running a few times.

[pocmo]

Is it really intermittent for the fennecProduction build variant though? We haven't had a nightly build of that in three days and it's starting to block our ability to test migration. :)

[ekager]

And today it's failing on
https://tools.taskcluster.net/groups/Szio0zDxTWiJ6sJIYkWULw/tasks/GxJU1i6jQvOSYdMvw73ZOA/runs/0/logs/public%2Flogs%2Flive.log

SUITE: org.mozilla.fenix.utils.DeleteAndQuitTest
  TEST: delete everything and quit
    [Robolectric] org.mozilla.fenix.utils.DeleteAndQuitTest.delete everything and quit: sdk=28; resources=LEGACY
    [Robolectric] NOTICE: legacy resources mode is deprecated; see http://robolectric.org/migrating/#migrating-to-40
    I/AssetManager: no overlays.list file found
    I/MonitoringInstr: Instrumentation started!
    I/MonitoringInstr: Setting context classloader to 'org.robolectric.internal.AndroidSandbox$SdkSandboxClassLoader@70ed0407', Original: 'org.robolectric.internal.AndroidSandbox$SdkSandboxClassLoader@70ed0407'
  FAILURE

java.lang.AssertionError: Verification failed: call 7 of 7: HomeActivity(#266).finish()) was not called.

Calls to same mock:
1) HomeActivity(#266).getApplicationContext()
2) HomeActivity(#266).getApplicationContext()
3) HomeActivity(#266).getApplicationContext()
4) HomeActivity(#266).getApplicationContext()
5) HomeActivity(#266).getApplicationContext()
6) HomeActivity(#266).getApplicationContext()
7) HomeActivity(#266).getApplicationContext()

[ekager]

Any idea why FennecProduction builds would run differently here?

[ekager]

I also can't reproduce this locally when running testGeckoBetaFennecProductionUnitTest 🤔
Can anyone see this fail locally ?

[pocmo]

Just saw that taskcluster runs test instead of the test for the flavor version (@mitchhentges That's probably an optimization we should do to reduce the time this runs).

The full command taskcluster runs:
./gradlew --no-daemon -PversionName=\"Signed-as-Fennec Nightly 190919 18:00\" clean test assembleFennecProduction

I wonder if we can reproduce this with that (maybe without the clean to save some time). Since this runs the tests multiple times for different flavors, maybe that breaks because of a previous run?

[mitchhentges]

I've created #5454 to do that optimization

[stale]

See: #17373 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.