Error status get reported when a browser fails to start initially but works OK on second try

[pkozlowski-opensource]

With mobile browsers (mostly IPhone on iOS) I can quite often see the following situation:

(1) Karma tries to start a browser. This initial start fails with:

ERROR [launcher.sauce]: Can not start iphone 7.1 (OS X 10.9)

[get("http://localhost:9876/?id=87195675")] Unexpected data in simpleCallback.

ERROR Job xxx is not in progress. It may have recently finished, or experienced an error. You can learn more at https://saucelabs.com/jobs/xxx

(2) This is fine, Karma tries to start the same browser again:

INFO [launcher]: Trying to start iphone 7.1 (OS X 10.9) on SauceLabs again (1/2).

and this time a mobile browser starts OK and tests are passing just fine.

Now the problem: when I inspect what gets reported to SauceLabs is the same status twice: once as Error and second time as Pass. Funnily enough SauceLabs takes the first (Error) result into account when displaying its badge so we've got IPhone build marked as errored even if it passed just fine.

I'm not super-clear where the actual problem should be fixed: Karma reporting or Sauce badge display but since we've got @ChrisWren wearing 2 hats I figured I will start here. On my side I will dig into the code but if anyone has ideas / input, it would be very much appreciated.

[ChrisWren]

@pkozlowski-opensource I have noticed something similar with selenium tests, did you by any chance make a ticket about the badge not updating? If not, I can chat with our support team.

[pkozlowski-opensource]

@ChrisWren nope, I didn't open any support ticket on the SauceLabs end as I think that we might want to improve things on the Karma end. But thinking about it some more - fix on Karma side won't help with other test runners, so yes, IMO it should be "fixed" on the Sauce end.

If you ask me the logic should be: take the latest result for the given build id (then again, I'm not even sure if the notation of the "latest" exists on the Sauce ends).

While we at the badges topic - the biggest pain atm is the fact that we can't distinguish the main branch builds vs. PR builds so if someone opens a PR that breaks a build we will see the broken status for the whole project... but yeh, this is a separate topic...

[ChrisWren]

@pkozlowski-opensource I think the issue on the sauce side is there is a duplicate entry in the db for a browser-platform-build combo so it is taking the result of the first one rather than the later one. I will make a ticket to see if we can use the latest browser-platform-build combo in the badge.

I agree that having a branch param for the badge would be helpful as well. We don't have a concept of branches as a top-level config property so it would have to be based on a tag or custom-data attribute.

[ChrisWren]

@pkozlowski-opensource I made a ticket internally and someone is working on it, will close when the fix is out

[pkozlowski-opensource]

@ChrisWren awesome! is the ticket for the double-reporting or for the no-badge-reporting for branch as well? I totally agree that even without a concept of a branch we could have a custom data attribute like "badge: true" or something. Or maybe it more belongs to the optional features? https://docs.saucelabs.com/reference/test-configuration/#optional-features

[ChrisWren]

@pkozlowski-opensource ticket is for the double-reporting, made one for branch reporting but we haven't started work on it yet.

[thom4parisot]

Are you sure it is mobile only? I encounter that error since a couple of days with all the browsers which used to work in the past: Safari 6, Safari 7, Chrome etc.

https://travis-ci.org/bbcrd/peaks.js/builds/29172710

[wswebcreation]

Closing this due to inactivity, feel free to re-open if this is still an issue.