We found 18 tets flaky

[shaker-project]

We run your test cases for 50 times and found that the following tests are flaky [1] (i.e., they pass or fail non-deterministically):

• openBookmarksPanel_showEmptyBookmarksView
• turboMode
• browsingWebsiteAndDeleteItsHistoryItem_deleteSuccessfully
• clearCache
• openLinkInNewTab
• typeTextInSearchFieldAndClear_textIsClearedAndBackToHome
• browsingTwoWebSites_sitesAreDisplayedInOrderInHistoryPanel
• editBookmarkWithClearingLocationContent_saveButtonIsDisabled
• addBookmarkAndEdit_bookmarkIsUpdated
• editBookmarkWithVariousWords_bookmarkIsUpdated
• shareWithFriends
• switchSearchEngine_searchViaSearchEngineAccordingly
• restoreEmptyTab_addNewTabThenRelaunch
• appHasOneTabAndReceiveBrowsingIntent_tabIncreasedAndBrowse
• browsingWebsiteBackAndForward_backAndFrowardToWebsite
• showRateAppPromotionAndClickFeedback_openFeedbackUrl
• clickSearchSuggestion_browseByDefaultSearchEngine
• deleteTopSite_deleteSuccessfully

We ran these tests on a generic Android Emulator (AVD) with Android API version 28.

To reproduce failures on these tests, please run the following script on gist:
https://gist.github.com/shaker-project/6c8c7bad2a4d96d24cec3746b9bab144
If you didn't understand any point of the script we can explain.

We require that stress-ng is installed. If not, run $> sudo apt-get install stress-ng.

If you confirm this behavior, our suggestion is to add the annotation @testflaky on these tests. See https://developer.android.com/reference/androidx/test/filters/FlakyTest

If you wish for a repair proposal, please let us know.

[1] https://hackernoon.com/flaky-tests-a-war-that-never-ends-9aa32fdef359

[Daisy-pliu]

Thanks for raising out.
In Firefox Lite , the tests we mark @flaky means it failed from time to time, they are flaky on our CI so we won't run there whenever source code changes, instead we will run those flaky tests manually before we release a build.
For those tests not marked @flaky they are running on our Bitrise CI Instrumentation Tests whenever new PR or PR merged. It runs well on emulators (e.g. NexusLowRes, 23 , en,portrait and NexusLowRes,24,en,portrait) for a long time. If there are failures we will take it down and try fixing it.
The priority for this issue might not be the highest at this moment in our roadmap but we can find some better timing to have a look into it if you would like to provide PR for it. Many thanks .