Breaking change for iPhone Simulators caused by node-saucelabs version bump

[web-padawan]

Hi, I have an issue related to the Sauce Labs update submitted at #206, landed in #207 and included to 4.1.5 release.

In my config I have the following launchers:

  'sl-safari-latest': {
    base: 'SauceLabs',
    browserName: 'safari',
    platform: 'macOS 10.13',
    version: 'latest'
  },
  'sl-ios-13': {
    base: 'SauceLabs',
    browserName: 'iphone',
    platform: 'iPhone X Simulator',
    version: '13.0'
  }

With 4.1.4 these worked fine. And when I upgrade to 4.1.5 Safari still works, but iPhone Simulator fails with an error:

12 10 2020 20:47:44.803:ERROR [SaucelabsLauncher]: Error: Failed to create session.
Misconfigured -- Unsupported OS/browser/version/device combo: OS: 'iPhone X Simulator', Browser: 'iPhone 11', Version: '13.0.', Device: 'unspecified'

These are the lines where the capabilities are updated so I'm wondering if something is missing from there?

@christian-bromann could you please check if this is an actual bug which needs to be fixed?
Or maybe something is wrong with my configuration for iPhone Simulator and it needs to be updated?

UPD: I also tried the config suggested by Platform Configurator but still no luck.

[christian-bromann]

@web-padawan I recommend to use W3C valid capabilities. The error message suggests that your capabilities are invalid.

[web-padawan]

Thanks. I would appreciate an example of valid W3C capabilities for iPhone Simulators on SauceLabs.

So far I have tried the following combinations:

1. platformName and platformVersion

  'sl-ios-13': {
    base: 'SauceLabs',
    browserName: 'Safari',
    platformName: 'iPhone X Simulator',
    platformVersion: '13.0'
  }

Result:

Error: Misconfigured -- Unsupported OS/browser/version/device combo: OS: 'iPhone X Simulator', Browser: 'safari', Version: 'latest', Device: 'unspecified'

2. platformName: 'iOS'

  'sl-ios-13': {
    base: 'SauceLabs',
    browserName: 'Safari',
    platformName: 'iOS',
    platformVersion: '13.0'
  }

Result:

There is no device that matches your criteria. DesiredCapabilities: {sauce:options={commandTimeout=600.0, customData={}, idleTimeout=600.0, maxDuration=5400.0, name=Lit Mixins Unit Tests - CI, public=public, recordScreenshots=false, recordVideo=false, tags=[], tunnelIdentifier=karma-sauce-1602527283}, browserName=Safari, platformName=iOS, platformVersion=13.0, browserVersion=latest, testobject_platform_name=iOS}

BTW, the docs here don't mention iOS Simulators: https://wiki.saucelabs.com/display/DOCS/W3C+Capabilities+Support

[wswebcreation]

@web-padawan

Try this

{
	browserName:'Safari',
	deviceName:'iPhone X Simulator',
	platformVersion:'13.0',
	platformName:'iOS',
	appiumVersion:'1.17.1',
	deviceOrientation:'portrait',
}

This is also according to the platform configurator. W3C caps are not yet fully implemented for Sauce Emulators/Simulators

[web-padawan]

Thanks, I have tried and got the same error:

Misconfigured -- Unsupported OS/browser/version/device combo: OS: 'iOS', Browser: 'safari', Version: 'latest', Device: 'unspecified'

One thing which confuses me is Device: 'unspecified' part even though deviceName is defined.

[christian-bromann]

@web-padawan can you provide a minimal reproducible example? I don't get any issues running these capabilities.

[web-padawan]

Here is the example: https://github.com/vaadin/component-mixins/compare/karma-sauce-issue

Failing build for this config: https://travis-ci.org/github/vaadin/component-mixins/builds/735138095

[christian-bromann]

@web-padawan we have reported the issue to the internal teams at Sauce. In the meantime stay on v4.1.4. I will keep you posted.

[web-padawan]

Thank you for prompt feedback! 🙇
This issue is also relevant for @web/test-runner-saucelabs which also uses W3C API: modernweb-dev/web#472 (comment)

[wswebcreation]

@web-padawan

This is fixed in this release, can you check https://github.com/karma-runner/karma-sauce-launcher/releases/tag/v4.3.1

[web-padawan]

Thank you, I tested and it works, my tests with iPhone Simulator are passing now 🙇‍♂️