Improve testdroid download test reliability

[mhutti1]

Sometimes the download test fails on testdroid. I think this is due to a connection problem. We should fix this.

[kelson42]

@mhutti1 We talk about ZIM download? If yes, this is a problem probably with either our distribution solution or by TestDroid. We should get the logs about this failing and act accordingly.

[julianharty]

I'm doing some analysis of the test failures on TestDroid. This is the most common test to fail and in the most recent failures has happened every time the test run has an error 13 times in the last 25 test runs. The other test that fails is the NetworkTest which has happened twice, a 2 in 25 error rate.

android.support.test.espresso.PerformException: Error performing 'load adapter data' on view 'with id: org.kiwix.kiwixmobile:id/library_list'

The line where the failure occurs is: org.kiwix.kiwixmobile.tests.DownloadTest.downloadTest(DownloadTest.java:113)

Our tests have only been run on one device for a long time. This is a Samsung S7 device running Android 7.0. The full model name is Samsung Galaxy S7 SM-G930F -BO
I've attached an example recording of when this error is reported.
record-9885e63336354e3744-1521030097.mp4.zip The download seemed to work. In this case although we download a ZIM with what appears to have a Cyrillic name: Авикипедиа (perhaps it's the ZIM for https://ab.wikipedia.org/wiki/%D0%90%D0%B2%D0%B8%D0%BA%D0%B8%D0%BF%D0%B5%D0%B4%D0%B8%D0%B0 ?) the downloaded file is in English and a Test ZIM with 3 articles in it of 348KB according to Kiwix. I don't currently understand why we'd download this file at all, nor how we end up with the test ZIM instead.

Occasionally we get a different error reported, still at the same line of code in DownloadTest.java (113)

android.support.test.espresso.PerformException: Error performing 'single click - At Coordinates: 539, 614 and precision: 16, 16' on view ' displaying data matching: with content 'ray_charles' within adapter view matching: with id: org.kiwix.kiwixmobile:id/library_list'.

In this test run the test selected Wikipedia to download rather than the usual much smaller target file. The download continues for several minutes according to the video of the test run (attached)
record-9885e63336354e3744-1521724815.mp4.zip

I'm guessing the first test might have timed-out, but didn't detect or mention the download was still continuing. This download was still running when the NetworkTest was running and perhaps affected the test's ability to complete successfully.

Let me try and sum up what I think I understand:

• Just over half the test runs fail with the DownloadTest.
• Two also failed with the NetworkTest, perhaps because the DownloadTest's download was still running as it was trying to download what seems to be the full Wikipedia.
• The videos help in identifying the second problem but not the first.
• We seem to download one file and find another. I don't yet understand why or what causes / triggers this to happen. Something to investigate as part of addressing this problematic test (and possibly the app?)

@ISNIT0 and I have started looking at the source code for DownloadTest. I'll add some comments when we've got a bit further in that process.

[julianharty]

Some more clues, at least on why the test sometimes downloads the strange file with the Cyrillic name... In the videos I notice there are no files listed for the Selected Languages. The file that gets downloaded is the first in the Other Languages.

In the device log there's a section that lists various parameters including the device model and locale. Here's an example. They seem pretty similar:

03-09 20:55:09.648 21555 21555 V d       : getOsVersion|value:7.0
03-09 20:55:09.648 21555 21555 V d       : getOsVersion|value:7.0
03-09 20:55:09.649 21555 21555 D [ACT]:S : Registering hardware receiver
03-09 20:55:09.651 21555 21555 D [ACT]:S : Loading the transmission policy
03-09 20:55:09.657 21555 21555 D [ACT]:W : Reading offline kvp file.
03-09 20:55:09.663 21555 21555 V c       : getNetworkCost|value:Unknown
03-09 20:55:09.663 21555 21555 V a       : getPowerSource|value:UNKNOWN
03-09 20:55:09.663 21555 21555 D [ACT]:ao: startProcessingWithTransmitCondition : UNMETERED_AC, profile: REAL_TIME
03-09 20:55:09.664 21555 21555 V d       : getAppId|value:
03-09 20:55:09.664 21555 21555 V af      : setAppId|appId: 
03-09 20:55:09.664 21555 21555 V d       : getAppVersion|value:16.0.9001.2077
03-09 20:55:09.664 21555 21555 V af      : setAppVersion|appVersion: 16.0.9001.2077
03-09 20:55:09.664 21555 21555 V a       : getDeviceId|value: 347e0831d733a1d1
03-09 20:55:09.664 21555 21555 V af      : setDeviceId|deviceId: 347e0831d733a1d1
03-09 20:55:09.664 21555 21555 V a       : getManufacturer|value: samsung
03-09 20:55:09.664 21555 21555 V af      : setDeviceMake|deviceMake: samsung
03-09 20:55:09.664 21555 21555 V a       : getModel|value: SM-G930F
03-09 20:55:09.664 21555 21555 V af      : setDeviceModel|deviceModel: SM-G930F
03-09 20:55:09.664 21555 21555 V c       : getNetworkProvider|value:
03-09 20:55:09.664 21555 21555 V af      : setNetworkProvider|networkProvider: 
03-09 20:55:09.664 21555 21555 V d       : getUserLanguage|value:en-US
03-09 20:55:09.664 21555 21555 V af      : setUserLanguage|language: en-US
03-09 20:55:09.664 21555 21555 V d       : getUserTimeZone|value:+01:00
03-09 20:55:09.664 21555 21555 V af      : setUserTimeZone|timeZone: +01:00
03-09 20:55:09.666 21555 21555 E ChinaFeaturesLib: failed to create chinaFeaturesHelper, ClassNotFoundException
03-09 20:55:09.666 21555 21555 D TSLTokenProviderImpl: Init

I've no idea what the ChinaFeaturesLib is or whether it's important. I searched for the error on Google and only found 5 matches, none with much insight as far as I can tell.

[julianharty]

BTW: in recent tests I see at least one instance (test run 1137) where we seem to get a different device, one with the locale set to en-GB rather than en-US. Again I don't yet know if it's significant. Here's the log extract that includes the strange ChinaFeaturesLib error BTW.

03-20 20:54:31.336 12041 12041 V d       : getOsVersion|value:7.0
03-20 20:54:31.336 12041 12041 V d       : getOsVersion|value:7.0
03-20 20:54:31.338 12041 12041 D [ACT]:S : Registering hardware receiver
03-20 20:54:31.340 12041 12041 D [ACT]:S : Loading the transmission policy
03-20 20:54:31.346 12041 12041 D [ACT]:W : Reading offline kvp file.
03-20 20:54:31.352 12071 12071 I FIPS_bssl: FIPS (v1.1) approved mode (1) | 12071 | top
03-20 20:54:31.352 12041 12041 V c       : getNetworkCost|value:Unknown
03-20 20:54:31.352 12041 12041 V a       : getPowerSource|value:UNKNOWN
03-20 20:54:31.352 12041 12041 D [ACT]:ao: startProcessingWithTransmitCondition : UNMETERED_AC, profile: REAL_TIME
03-20 20:54:31.353 12041 12041 V d       : getAppId|value:
03-20 20:54:31.353 12041 12041 V af      : setAppId|appId: 
03-20 20:54:31.353 12041 12041 V d       : getAppVersion|value:16.0.9001.2077
03-20 20:54:31.353 12041 12041 V af      : setAppVersion|appVersion: 16.0.9001.2077
03-20 20:54:31.353 12041 12041 V a       : getDeviceId|value: b8882617824f1bc9
03-20 20:54:31.353 12041 12041 V af      : setDeviceId|deviceId: b8882617824f1bc9
03-20 20:54:31.353 12041 12041 V a       : getManufacturer|value: samsung
03-20 20:54:31.353 12041 12041 V af      : setDeviceMake|deviceMake: samsung
03-20 20:54:31.353 12041 12041 V a       : getModel|value: SM-G930F
03-20 20:54:31.354 12041 12041 V af      : setDeviceModel|deviceModel: SM-G930F
03-20 20:54:31.354 12041 12041 V c       : getNetworkProvider|value:
03-20 20:54:31.354 12041 12041 V af      : setNetworkProvider|networkProvider: 
03-20 20:54:31.354 12041 12041 V d       : getUserLanguage|value:en-GB
03-20 20:54:31.354 12041 12041 V af      : setUserLanguage|language: en-GB
03-20 20:54:31.354 12041 12041 V d       : getUserTimeZone|value:+01:00
03-20 20:54:31.354 12041 12041 V af      : setUserTimeZone|timeZone: +01:00
03-20 20:54:31.356 12041 12041 E ChinaFeaturesLib: failed to create chinaFeaturesHelper, ClassNotFoundException
03-20 20:54:31.356 12041 12041 D TSLTokenProviderImpl: Init

[julianharty]

I see tests that pass with the locale of en-US e.g. 1085

03-09 13:18:38.580 11058 11058 V d       : getUserLanguage|value:en-US
03-09 13:18:38.580 11058 11058 V af      : setUserLanguage|language: en-US

[mhutti1]

This could be todo with #510 on certain locals specifically SSL certificate issues?

[mhutti1]

Sometimes we get

SSL handshake timed out

Which I think is a retrofit error.

[julianharty]

@mhutti1 As part of testing changes @ISNIT0 and I have been working on to improve the automated tests #246 I ran some large-scale tests in TestDroid (30 devices). 99.1% of the tests passed. Of the few that failed, several coincided with the WiFi connection dropping / reconnecting on the device. I'm going to ask the support team at Bitbar to investigate. I'll cc you on the email. I don't want to put the details here as they relate to their infrastructure.

[julianharty]

It's time to close this issue as our tests are now running reliably and successfully. The most recent failure was for testRightDrawer. The download test has passed on 3 devices for the last 50+ test runs on BitBar / TestDroid. When I saw failures they were reporting actual problems in the test infrastructure :)