browser tags should include version (and OS)

[ctaepper]

we need to test multiple browser version on multiple devices, e.g.

• Safari 10 on OSX
• Safari 11 on OSX
• Safari on iOS 10
• Safari on iOS 11

we use saucelabs for testing and all works well, screenshots and json reports are produced.

however, when generating the HTML reports, the tag "Browser" seems to get summed up be the multiCapabilities property browserName, which in all 4 cases is Safari. there doesn't seem to be an apperant way to define those tags in a custom way, at least not with the usage of mocha.

or am i missing something?

[jan-molak]

Hi there,

This feature hasn't been implemented for 2.0 yet.

• Since detecting the browser type needs to happen right after the scenario starts, which might be before there's an actor in the spotlight, this feature has a dependency on #362. Once this is done, browser-specific could be added to enable filtering in the test report.
• To display the OS icon as well as the browser icon in the test results lists, Serenity BDD will have to support more than one "context icon" per test scenario. This PR should address that serenity-bdd/serenity-core#1860
  

Thanks for your patience and watch this space!

[jan-molak]

Hi @ctaepper - so I'm thinking about the browser tags you mentioned.

Let's consider the 4 scenarios:

1. Safari 10 on OSX
2. Safari 11 on OSX
3. Safari on iOS 12
4. Safari on iOS 13

What should be the tags you'd like to see in each scenario?

My initial thoughts are:

1. safari, safari-10, darwin, darwin-19.0.0
2. safari, safari-11, darwin, darwin-19.0.0
3. safari, safari-12.1, iOS, iOS-12.4
4. safari, safari-12.1, iOS, iOS-13.2

This should help to test the test results by browser, by specific browser version, by OS and by OS version.

How does that sound?

Jan

[ctaepper]

hm, so my initial thoughts were to retrieve this tags from the protractor multiCapabilities configuration, because this was my use case back then and still is today. This config looks something like this:

// protractor.conf.js
...
multiCapabilities: [
        // OSX Chrome 77
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'OS X',
            os_version: 'Mojave',
            browserName: 'Chrome',
            browser_version: '77.0',
            resolution: '1280x1024',
            project: 'legal-e2e'
        },
        // OSX Firefox 69
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'OS X',
            os_version: 'Mojave',
            browserName: 'Firefox',
            browser_version: '69.0',
            resolution: '1280x1024',
            project: 'legal-e2e'
        },
        // OSX Safari 12
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'OS X',
            os_version: 'Mojave',
            browserName: 'Safari',
            browser_version: '12.1',
            resolution: '1280x1024',
            project: 'legal-e2e'
        },
        // Windows 10 Chrome 77
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'Windows',
            os_version: '10',
            browserName: 'Chrome',
            browser_version: '77.0',
            resolution: '1280x1024',
            project: 'legal-e2e'
        },
        // Windows 10 Firefox 69
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'Windows',
            os_version: '10',
            browserName: 'Firefox',
            browser_version: '69.0',
            resolution: '1280x1024',
            project: 'legal-e2e'
        },
        // Windows 10 IE 11
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'Windows',
            os_version: '10',
            browserName: 'IE',
            browser_version: '11.0',
            resolution: '1280x1024',
            project: 'legal-e2e'
        },
        // Windows 10 Edge 18
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'Windows',
            os_version: '10',
            browserName: 'Edge',
            browser_version: '18.0',
            resolution: '1280x1024',
            project: 'legal-e2e'
        },
        // Windows 10 Edge preview
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'Windows',
            os_version: '10',
            browserName: 'Edge',
            browser_version: 'insider preview',
            resolution: '1280x1024',
            project: 'legal-e2e'
        }
    ],
...

both os and browser versions can be retrieved from there.

but this is protractor/selenium specific. and not a general approach.

The tags that you suggest look 100% good to me. I would be happy to have them exactly like this. However, it would still be great to somehow add custom tags per this multiCapabilities property, because only then I immediately know which configuration failed.

[jan-molak]

Cool, could you please elaborate a bit more on

it would still be great to somehow add custom tags per this multiCapabilities property, because only then I immediately know which configuration failed.

How do you think this should work? What custom tags would you like to see for the configurations you mention?

[jan-molak]

Hey @ctaepper - apologies you had to wait for a while for this to arrive.

Check out Serenity/JS 2.1.0 and let me know what you think!

---

You can now sponsor my work on Serenity/JS on Github