Intermittent UI test failure - MediaNotificationTest.videoPlaybackSystemNotificationTest

[jonalmeida]

Firebase Test Run:

https://console.firebase.google.com/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/4983241498387819676

Stacktrace:

• See the results at the link for the tests:
  • audioPlaybackSystemNotificationTest
  • mediaSystemNotificationInPrivateModeTest
  • tabMediaControlButtonTest

Build:

• AC upgrade and prior: Upgrade to Android Component 62.0.20201006190820 with breaking changes #15734

**All media tests are now re-enabled, except the videoPlaybackSystemNotificationTest which is still waiting for mozilla-mobile/android-components#9748

┆Issue is synchronized with this Jira Task

[jonalmeida]

Copied over from: #15734 (comment)

---

There are a few tests that are failing, so I ended up testing with tabMediaControlButtonTest from the following list:

• audioPlaybackSystemNotificationTest - apparent intermittent; consistent failing now
• mediaSystemNotificationInPrivateModeTest - hard to reproduce; need older API level device that I do not have.
• tabMediaControlButtonTest - should be consistent.

I tried a bunch of GV builds to see what the media failure is but I've come across some confusing results. Using the latest AC (3e68528074) with these GV Nightly versions tested:

• 83.0.20201005093214 - ❌ (monday's GV after media regression was fixed)
• 83.0.20201001094020 - ❌ (GV media breakage)
• 83.0.20200930092918 - ❌ (before GV media breakage; expected to fail)
• 83.0.20200929094055 - ❌ (second before media breakage; expected to fail)

GV media regression was fixed (feature was disabled) in 20201001094020 so the 83.0.20201005093214 should have worked.

• https://hg.mozilla.org/mozilla-central/rev/e4e35e1aae77

[jonalmeida]

Adding videoPlaybackSystemNotificationTest to the list of ignored tests which seem to fail (intermittently though).

[jonalmeida]

To expand on comment 2, I was able to see audioPlaybackSystemNotificationTest consistently although it was called out as an intermittent failure.

You can read more about that test analysis here: #15734 (comment)

---

For mediaSystemNotificationInPrivateModeTest it seems like it asserts on the Android OS system media notification which was hard for me to trust with my Android 11 device that has a new UI design for the system notifications - for that reason I didn't look further into why that test was failing.

---

For tabMediaControlButtonTest, previously we saw this test legitimately fail and catch a regression. Visually, this test was seen trying to press the play button and that would fail.

With the regression fixed, we were able to land a fixed GV version. Now however, we are seeing the same test fail, but visually watching the test run, we can see the media content play, but the assertion always fails.

[sv-ohorvath]

Tried to fix the tests but found there is an open bug blocking media notifications for some videos: mozilla-mobile/android-components#9748. We need to wait for that to be fixed.

[sv-ohorvath]

All media tests are now re-enabled, except the videoPlaybackSystemNotificationTest which is still waiting for mozilla-mobile/android-components#9748

[stale]

See: #17373 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sv-ohorvath]

Lets keep it.

[sv-ohorvath]

After a bit more investigation I've found that videos muted or without sound don't show a notification, I left a new comment on mozilla-mobile/android-components#9748.
The video asset used in this test didn't have sound so I replaced it with a new file which now looks like it could fix the problem, but while running on Firebase I hit the same ANR reported here: #24047 so I'll leave it disabled until we find a fix for that.

Latest test runs (with ANR results): https://github.com/mozilla-mobile/fenix/runs/6707769666

[sv-ohorvath]

The tests were re-enabled in #27060