CI: Appium tests on Sauce all failing

[janpio]

All failing:

iOS:
https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514197
https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514198
https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514199
https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514200

Android:
https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514201
https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514202
https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514203
https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514204
https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514205
https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514206

---

Problem is that Paramedic itself doesn't have any tests of this kind, so it is difficult to test here in this repo first :/

[janpio]

I checked older, successful CI runs of cordova-plugin-camera, and it seems the Appium tests haven't been run for quite some time:

...
cordova-paramedic: tests have been completed
cordova-paramedic: Tests finished
Start running Appium tests...
Running Appium tests on Sauce Labs
Couldn't find Appium tests, skipping...
Completed tests at 5:21:17 PM
Getting saucelabs jobs details...
...

https://travis-ci.org/apache/cordova-plugin-camera/builds/528500727?utm_source=github_status&utm_medium=notification

Somehow "Couldn't find Appium tests, skipping..." got fixed in the last 5 days.

[janpio]

Could find a successful CI run with Appium tests actually being executed for the contacts plugin:
https://travis-ci.org/apache/cordova-plugin-contacts/jobs/491749438
3 months ago.

---

On further looking:

A Android run for the camera plugin that succeeded:
https://travis-ci.org/apache/cordova-plugin-camera/jobs/514613162
~1 month ago

Before that, ~2 months ago, we have successful Appium runs for iOS and Android:
https://travis-ci.org/apache/cordova-plugin-camera/jobs/507356980
https://travis-ci.org/apache/cordova-plugin-camera/jobs/507356981
https://travis-ci.org/apache/cordova-plugin-camera/jobs/507356982
https://travis-ci.org/apache/cordova-plugin-camera/jobs/507356983
https://travis-ci.org/apache/cordova-plugin-camera/jobs/507356984

---

So the question now is: What changed between these successful runs and now?

[janpio]

• We changed the OS versions the tests are run on (all new iOS versions as old simulators are gone)
• We added an environment for the tests to run on (local instead of just saucelabs)
• We changed the whole travis.yml structure (but this shouldn't influence things)
• We changed the OS version configuration details (saucelabs parameters like appiumVersion etc)
• We rewrote good parts of paramedic, that is used to actually run the tests

[janpio]

Similar OS/version combinations being run then and now are: Andorid 4.4, 5.1, 6.0, 7.0.
For these combinations debugging makes most sense, as theoretically it should still work.

[janpio]

Looking into the failures we are getting a bit, they look different at first, but turn out to be the same thing:

iOS:

- Failed: Failed to start a session

Android:

- ReferenceError: lastFailureReason is not defined

Turns out the Android error comes from this line:
https://github.com/apache/cordova-plugin-camera/blob/295e928784e2a9785982fbf15b05215a317774df/appium-tests/android/android.spec.js#L353

Here lastFailureReason is indeed undefined, it doesn't appear anywhere else. Otherwise this would also output:

Failed to start a session

Why though... no idea yet.

[janpio]

Also tracked at:
apache/cordova-plugin-camera#466
apache/cordova-plugin-camera#465

[janpio]

As a temporary workaround, the Appium tests were removed from camera. A new PR will add them back: apache/cordova-plugin-camera#469