Gordon does not pick up all available devices

[kubovy]

Hi, thanks for this tool. It delivers what it promises in regards of retrying tests, which I was missing in other tools before.

I am having a CI machine with one emulator on it and three connected devices. Another small computer with one emulator on it. And a second computer with 3 emulators on it. I tried using both, PoolingStrategy.SinglePool and PoolingStrategy.Manual with all devices in one pool. It seams that gordon groups the above per type (real, emulator) and per host. As soon as one of the groups has more than one device it deterministically skips one device.

For the above example: only CI Emulator, and 2 real devices will be picked, on the machine with one emulator, that one will be also picked. The machine with 3 emulators only 2 will be picked.

The interesting part is if I disconnect one real devices then only one of the remaining two real devices will be picked. Otherwise everything stays the same.

If I stop one of the 3 emulators on the same host, then only one of the remaining two emulators will be picked.

More interesting it is deterministic: always the same devices will be skipped:

Scenario 1:

real-device-1   online   NOT PICKED
real-device-2   online   picked
real-device-3   online   picked
emulator:5555   online   picked
host-a:7555     online   picked
host-b:8555     online   picked
host-b:8557     online   picked
host-b:8559     online   NOT PICKED

Scenario 2:

real-device-1   offline   
real-device-2   online   NOT PICKED
real-device-3   online   picked
emulator:5555   online   picked
host-a:7555     online   picked
host-b:8555     online   picked
host-b:8557     online   NOT PICKED
host-b:8559     offline   

Start or connection order of the devices/emulators does not make any difference. I never managed that all devices are picked.

There are no messages in the logs regarding timeouts or anything mentioning the skipped devices, they just don't appear in the log at all.

Any ideas?
Thanks!

[joshschriever]

@kubovy can you clarify what you're expecting? With all devices in one pool, each test will only run on one device. Do you have a small number of tests, such that not all devices are needed?

Or if you're expecting each test to run on each device, then what you want is PoolingStrategy.PoolPerDevice.

[kubovy]

@joshschriever thank you for the fast reaction. I understand the pooling strategies.

In my setup all devices are in one pool, e.g. with PoolingStrategy.SinglePool or PoolingStrategy.Manual (all devices manually added to a single pool) - tried both options to see if it makes any difference.

The scenarios I described above is that there are always devices (marked as NOT PICKED above) which are completely ignored during the whole test run. I have around 600 tests. They run (one test per device) only on the devices marked above as picked.

My expectation would be that all devices in the pool will be at some point used during the whole run. Running each test once on any of those devices.

More clarification: before the test starts I see in the log the installation of the app APK and test APK, already there the ignored devices are consistently not shown. No timeout message, etc. I also increased the installation timeout but no change.

... building part of the log ...
00:21:55.440  > Task :app:gordon
00:22:42.170  real-device-2: installing app.package.name
00:22:42.170  real-device-3: installing app.package.name
00:22:42.170  host-a:7555: installing app.package.name
00:22:42.170  host-b:8555: installing app.package.name
00:22:42.170  
00:22:42.170  The APKs have been extracted in the directory: /tmp/830506879766754033
00:22:42.170  The APKs have been extracted in the directory: /tmp/15378697174757927620
00:22:42.171  The APKs have been extracted in the directory: /tmp/14582370386234563643
00:22:42.171  The APKs have been extracted in the directory: /tmp/2033343059800847795
00:22:52.158  
00:22:52.158  > Task :app:gordon
00:22:52.158  host-b:8555: installing app.package.name.test
00:22:54.078  host-a:7555: installing app.package.name.test
00:22:57.383  host-b:8557: installing app.package.name
00:22:57.383  
00:22:57.383  The APKs have been extracted in the directory: /tmp/3186994027525455358
00:23:09.634  
00:23:09.634  > Task :app:gordon
00:23:09.634  host-b:8557: installing app.package.name.test
00:24:46.085  real-device-2: installing app.package.name.test
00:24:46.085  real-device-3: installing app.package.name.test
... tests start ....

In the log the devices: real-device-1, host-b:8559 do not show nor are those devices used.

As I said, if I turn off/disconnect the ignored devices another 2 devices will be not picked at all. In the example shown in the log if I disconnect real-device-1, real-device-2 will not be picked. If I turn off host-b:8559, host-b:8557 will not be picked by the test at all.

It seams strange. But it somehow seems as if the devices get clustered by type (real/emulator) and host where they run. The behavior described above seems to then apply for each cluster which has more than 1 device:

If I have: real-device-1, emulator-1, host-a:7555, host-b:8555 all devices get picked.
If I have: real-device-1, real-device-2, emulator-1, host-a:7555, host-b:8555 => real-device-1 does not get picked.
If I have: real-device-1, real-device-2, real-device-3, emulator-1, host-a:7555, host-b:8555 => real-device-1 does not get picked.
If I have: real-device-1, emulator-1, host-a:7555, host-b:8555, host-b:8557 => host-b:8557 does not get picked.
If I have: real-device-1, emulator-1, host-a:7555, host-b:8555, host-b:8557, host-b:8559 => host-b:8559 does not get picked.

Weird right?

[joshschriever]

That is weird. Seems like the way Gordon uses ADB is not picking up all the devices. I'll convert this discussion to an issue for investigation.

[joshschriever]

#117