Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

browser tags should include version (and OS) #132

Closed
ctaepper opened this issue Dec 16, 2017 · 5 comments
Closed

browser tags should include version (and OS) #132

ctaepper opened this issue Dec 16, 2017 · 5 comments
Labels
enhancement @serenity-js/core

Comments

@ctaepper
Copy link

@ctaepper ctaepper commented Dec 16, 2017

we need to test multiple browser version on multiple devices, e.g.

  • Safari 10 on OSX
  • Safari 11 on OSX
  • Safari on iOS 10
  • Safari on iOS 11

we use saucelabs for testing and all works well, screenshots and json reports are produced.

however, when generating the HTML reports, the tag "Browser" seems to get summed up be the multiCapabilities property browserName, which in all 4 cases is Safari. there doesn't seem to be an apperant way to define those tags in a custom way, at least not with the usage of mocha.

or am i missing something?

@jan-molak jan-molak added the enhancement label Jan 12, 2018
@jan-molak jan-molak mentioned this issue Feb 6, 2018
5 tasks
@jan-molak jan-molak added this to the 2.0 milestone Feb 6, 2019
@ctaepper ctaepper mentioned this issue Oct 22, 2019
@jan-molak
Copy link
Member

@jan-molak jan-molak commented Oct 30, 2019

Hi there,

This feature hasn't been implemented for 2.0 yet.

  • Since detecting the browser type needs to happen right after the scenario starts, which might be before there's an actor in the spotlight, this feature has a dependency on #362. Once this is done, browser-specific could be added to enable filtering in the test report.
  • To display the OS icon as well as the browser icon in the test results lists, Serenity BDD will have to support more than one "context icon" per test scenario. This PR should address that serenity-bdd/serenity-core#1860
    image

Thanks for your patience and watch this space!

@jan-molak
Copy link
Member

@jan-molak jan-molak commented Nov 4, 2019

Hi @ctaepper - so I'm thinking about the browser tags you mentioned.

Let's consider the 4 scenarios:

  1. Safari 10 on OSX
  2. Safari 11 on OSX
  3. Safari on iOS 12
  4. Safari on iOS 13

What should be the tags you'd like to see in each scenario?

My initial thoughts are:

  1. safari, safari-10, darwin, darwin-19.0.0
  2. safari, safari-11, darwin, darwin-19.0.0
  3. safari, safari-12.1, iOS, iOS-12.4
  4. safari, safari-12.1, iOS, iOS-13.2

This should help to test the test results by browser, by specific browser version, by OS and by OS version.

How does that sound?

Jan

@ctaepper
Copy link
Author

@ctaepper ctaepper commented Nov 4, 2019

hm, so my initial thoughts were to retrieve this tags from the protractor multiCapabilities configuration, because this was my use case back then and still is today. This config looks something like this:

// protractor.conf.js
...
multiCapabilities: [
        // OSX Chrome 77
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'OS X',
            os_version: 'Mojave',
            browserName: 'Chrome',
            browser_version: '77.0',
            resolution: '1280x1024',
            project: 'legal-e2e'
        },
        // OSX Firefox 69
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'OS X',
            os_version: 'Mojave',
            browserName: 'Firefox',
            browser_version: '69.0',
            resolution: '1280x1024',
            project: 'legal-e2e'
        },
        // OSX Safari 12
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'OS X',
            os_version: 'Mojave',
            browserName: 'Safari',
            browser_version: '12.1',
            resolution: '1280x1024',
            project: 'legal-e2e'
        },
        // Windows 10 Chrome 77
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'Windows',
            os_version: '10',
            browserName: 'Chrome',
            browser_version: '77.0',
            resolution: '1280x1024',
            project: 'legal-e2e'
        },
        // Windows 10 Firefox 69
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'Windows',
            os_version: '10',
            browserName: 'Firefox',
            browser_version: '69.0',
            resolution: '1280x1024',
            project: 'legal-e2e'
        },
        // Windows 10 IE 11
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'Windows',
            os_version: '10',
            browserName: 'IE',
            browser_version: '11.0',
            resolution: '1280x1024',
            project: 'legal-e2e'
        },
        // Windows 10 Edge 18
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'Windows',
            os_version: '10',
            browserName: 'Edge',
            browser_version: '18.0',
            resolution: '1280x1024',
            project: 'legal-e2e'
        },
        // Windows 10 Edge preview
        {
            'browserstack.user': process.env.BROWSERSTACK_USER,
            'browserstack.key': process.env.BROWSERSTACK_KEY,
            'browserstack.local': true,
            os: 'Windows',
            os_version: '10',
            browserName: 'Edge',
            browser_version: 'insider preview',
            resolution: '1280x1024',
            project: 'legal-e2e'
        }
    ],
...

both os and browser versions can be retrieved from there.

but this is protractor/selenium specific. and not a general approach.

The tags that you suggest look 100% good to me. I would be happy to have them exactly like this. However, it would still be great to somehow add custom tags per this multiCapabilities property, because only then I immediately know which configuration failed.

@jan-molak
Copy link
Member

@jan-molak jan-molak commented Nov 4, 2019

Cool, could you please elaborate a bit more on

it would still be great to somehow add custom tags per this multiCapabilities property, because only then I immediately know which configuration failed.

How do you think this should work? What custom tags would you like to see for the configurations you mention?

jan-molak added a commit that referenced this issue Nov 10, 2019
…easier to distinguish

The filename of a screenshot contains the name of the OS, browser and the browser version, as well
as the name of the interaction the screenshot has been captured for. The filename of a test report
contains the name of the scenario.

Enabler for #132
@jan-molak jan-molak removed the ready label Feb 1, 2020
@jan-molak
Copy link
Member

@jan-molak jan-molak commented Feb 7, 2020

Hey @ctaepper - apologies you had to wait for a while for this to arrive.

Check out Serenity/JS 2.1.0 and let me know what you think!

image


You can now sponsor my work on Serenity/JS on Github :octocat:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement @serenity-js/core
Projects
None yet
Development

No branches or pull requests

2 participants