Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Improve testdroid download test reliability #283
I'm doing some analysis of the test failures on TestDroid. This is the most common test to fail and in the most recent failures has happened every time the test run has an error 13 times in the last 25 test runs. The other test that fails is the NetworkTest which has happened twice, a 2 in 25 error rate.
The line where the failure occurs is:
Our tests have only been run on one device for a long time. This is a Samsung S7 device running Android 7.0. The full model name is
Occasionally we get a different error reported, still at the same line of code in DownloadTest.java (113)
In this test run the test selected Wikipedia to download rather than the usual much smaller target file. The download continues for several minutes according to the video of the test run (attached)
I'm guessing the first test might have timed-out, but didn't detect or mention the download was still continuing. This download was still running when the NetworkTest was running and perhaps affected the test's ability to complete successfully.
Let me try and sum up what I think I understand:
@ISNIT0 and I have started looking at the source code for
Some more clues, at least on why the test sometimes downloads the strange file with the Cyrillic name... In the videos I notice there are no files listed for the
In the device log there's a section that lists various parameters including the device model and locale. Here's an example. They seem pretty similar:
I've no idea what the
BTW: in recent tests I see at least one instance (test run 1137) where we seem to get a different device, one with the locale set to
referenced this issue
Apr 1, 2018
@mhutti1 As part of testing changes @ISNIT0 and I have been working on to improve the automated tests #246 I ran some large-scale tests in TestDroid (30 devices). 99.1% of the tests passed. Of the few that failed, several coincided with the WiFi connection dropping / reconnecting on the device. I'm going to ask the support team at Bitbar to investigate. I'll cc you on the email. I don't want to put the details here as they relate to their infrastructure.
It's time to close this issue as our tests are now running reliably and successfully. The most recent failure was for