Is there a way to set context tags for multi-browser testing?

[robin-weller]

Hi Jan,

With the Java Serenity version, you can provide a context for each browser so that the report shows which browser the test has run on. I have found an old example on John Smart's blog and it still works with the latest version.

Is this something that can easily be done with Serenity/JS?

I am able to run the tests in multiple browsers but there is no distinction in the report to show which browser the test run on.

[abhinaba-ghosh]

@robin-weller
Actually serenity JS report does show the browser information the nicest way possible. I am working on serenity 1.x version and able to get below information in the report:

Not only that, for multi-browser tests, all the browsers will be listed down here.

Most important thing is, before each test case, it shows the icon in which browser tests are executed.

Can you share your protractor.conf and the report you are getting. It will ravel a broader picture.

[robin-weller]

That is what I see using the Java version. I am using v2 though.

protractor.conf.js:

const
    { ConsoleReporter } = require('@serenity-js/console-reporter'),
    { ArtifactArchiver } = require('@serenity-js/core'),
    { SerenityBDDReporter } = require('@serenity-js/serenity-bdd'),
    { Photographer, TakePhotosOfInteractions } = require('@serenity-js/protractor');

exports.config = {
    chromeDriver: require('chromedriver/lib/chromedriver').path,
    geckoDriver: require('geckodriver/lib/geckodriver').path,
    SELENIUM_PROMISE_MANAGER: false,

    directConnect: true,

    allScriptsTimeout: 11000,

    specs: [ './features/*.feature', ],

    framework: 'custom',
    frameworkPath: require.resolve('@serenity-js/protractor/adapter'),

    onPrepare: () => {
        require('ts-node').register({
            project: './tsconfig.json'
        });
        browser.waitForAngularEnabled(false);
    },

    serenity: {
        crew: [
            ArtifactArchiver.storingArtifactsAt('./target/site/serenity'),
            Photographer.whoWill(TakePhotosOfInteractions),
            new SerenityBDDReporter(),
            ConsoleReporter.forDarkTerminals(),
        ],
        runner: 'cucumber'
    },

    cucumberOpts: {
        require: [
            'features/steps/**/*.ts',
            'features/support/setup.ts',
        ],
        'require-module': ['ts-node/register'],
    },

    multiCapabilities: [
        {
            browserName: "chrome",
            chromeOptions: {
                args: [
                    '--disable-infobars',
                    '--no-sandbox',
                    '--disable-gpu',
                    '--window-size=1024x768',
                    // '--headless',
                ],
            },
        },
        {
            browserName: "firefox",
        },
    ],
};

Report:

A Java version:

Edit: In the java version, I set -Dcontext=browserName but not sure how I would set this?

[abhinaba-ghosh]

@jan-molak Is it related to old serenity CLI jar?

[jan-molak]

Hi @robin-weller, it's great to hear from you again!

@abhinaba1080 - your instincts are right, it is partially related to the old Serenity CLI jar. The upcoming 2.1.10 release of the jar will come with serenity-bdd/serenity-core#1860, which enables both browser and OS icons to be rendered for every test scenario.

I was waiting for that jar to be published to add both the browser/os tagging as well as the icons (#132) all in one go. But, having thought about it some more, I decided to implement the tagging first and the icons whenever the jar arrives.

Enjoy Serenity/JS v 2.1.0 with OS and browser tags

Support for icons is coming soon ;-)

---

You can now sponsor my work on Serenity/JS on Github

[robin-weller]

Thanks @jan-molak - good to hear from you too!

I am seeing the OS and browser tags in the report, however, there is an issue generating the report from firefox runs. so it is only showing chrome.

I see the following error when it is generating reports from firefox runs:
[main] WARN n.t.c.r.json.JSONTestOutcomeReporter - This file was not a valid JSON Serenity test report: scenario-user-can-log-in-ec36ba7706.json

I am seeing the chrome icon though!

[jan-molak]

Cool, so we're making progress :-) Would you mind posting the offending json file?

[robin-weller]

Of course:

{ 
   "name":"User can log in",
   "title":"User can log in",
   "id":"demo-journeys;user-can-log-in",
   "manual":false,
   "testSteps":[ 
      { 
         "number":1,
         "description":"Given Arya is on the <Obfuscated Website> homepage",
         "startTime":1581086694376,
         "children":[ 
            { 
               "number":2,
               "description":"Arya navigates to '<Obfuscated Website>'",
               "startTime":1581086694379,
               "children":[ 

               ],
               "reportData":[ 

               ],
               "screenshots":[ 
                  { 
                     "screenshot":"photo-firefox-arya-navigates-to-<Obfuscated Website>-cf8036b9c1.png"
                  }
               ],
               "result":"SUCCESS",
               "duration":905
            },
            { 
               "number":3,
               "description":"Arya ensures that the title of the current page does start with '<Obfuscated Website>'",
               "startTime":1581086695287,
               "children":[ 

               ],
               "reportData":[ 

               ],
               "screenshots":[ 
                  { 
                     "screenshot":"photo-firefox-arya-ensures-that-the-title-of-the-current-page-does-start-with-<Obfuscated Website>-b2144a1f3b.png"
                  }
               ],
               "result":"SUCCESS",
               "duration":5
            }
         ],
         "reportData":[ 

         ],
         "screenshots":[ 

         ],
         "result":"SUCCESS",
         "duration":919
      },
      { 
         "number":4,
         "description":"When she logs in",
         "startTime":1581086695296,
         "children":[ 
            { 
               "number":5,
               "description":"Arya logs in to <Obfuscated Website>",
               "startTime":1581086695297,
               "children":[ 
                  { 
                     "number":6,
                     "description":"Arya clicks on the the user icon",
                     "startTime":1581086695297,
                     "children":[ 

                     ],
                     "reportData":[ 

                     ],
                     "screenshots":[ 
                        { 
                           "screenshot":"photo-firefox-arya-clicks-on-the-the-user-icon-dad1f72b8e.png"
                        }
                     ],
                     "result":"SUCCESS",
                     "duration":2581
                  },
                  { 
                     "number":7,
                     "description":"Arya enters '<Obfuscated Email>' into the the email address field",
                     "startTime":1581086697879,
                     "children":[ 

                     ],
                     "reportData":[ 

                     ],
                     "screenshots":[ 
                        { 
                           "screenshot":"photo-firefox-arya-enters-<Obfuscated Email>-into-the-the-email-address-field-1bd27fbedd.png"
                        }
                     ],
                     "result":"SUCCESS",
                     "duration":412
                  },
                  { 
                     "number":8,
                     "description":"Arya enters '<Obfuscated Password>' into the the password field",
                     "startTime":1581086698292,
                     "children":[ 

                     ],
                     "reportData":[ 

                     ],
                     "screenshots":[ 
                        { 
                           "screenshot":"photo-firefox-arya-enters-<Obfuscated Password>-into-the-the-password-field-3673e82b69.png"
                        }
                     ],
                     "result":"SUCCESS",
                     "duration":182
                  },
                  { 
                     "number":9,
                     "description":"Arya clicks on the the sign in button",
                     "startTime":1581086698476,
                     "children":[ 

                     ],
                     "reportData":[ 

                     ],
                     "screenshots":[ 
                        { 
                           "screenshot":"photo-firefox-arya-clicks-on-the-the-sign-in-button-68499d3cfd.png"
                        }
                     ],
                     "result":"SUCCESS",
                     "duration":513
                  }
               ],
               "reportData":[ 

               ],
               "screenshots":[ 

               ],
               "result":"SUCCESS",
               "duration":3693
            }
         ],
         "reportData":[ 

         ],
         "screenshots":[ 

         ],
         "result":"SUCCESS",
         "duration":3695
      },
      { 
         "number":10,
         "description":"Then she should see that she is logged in",
         "startTime":1581086698991,
         "children":[ 

         ],
         "reportData":[ 

         ],
         "screenshots":[ 

         ],
         "result":"PENDING",
         "duration":1
      }
   ],
   "userStory":{ 
      "id":"demo-journeys",
      "storyName":"Demo Journeys",
      "path":"features/demo.feature",
      "type":"feature"
   },
   "startTime":1581086694369,
   "testSource":"Cucumber",
   "tags":[ 
      { 
         "name":"Demo Journeys",
         "type":"feature"
      },
      { 
         "name":"firefox ",
         "type":"browser",
         "browserName":"firefox"
      },
      { 
         "type":"platform"
      }
   ],
   "featureTag":{ 
      "name":"Demo Journeys",
      "type":"feature"
   },
   "context":"firefox",
   "driver":"firefox",
   "duration":4811,
   "result":"PENDING"
}

[robin-weller]

The difference I am seeing between the Chome and Firefox JSON files is there is no recorded browserVersion for Firefox (which is probably fine?!) and there is no name defined with "type": "platform"?

One final thing:
Chrome:

{
        "name": "chrome 80.0.3987.87"
        , "type": "browser"
        , "browserName": "chrome"
        , "browserVersion": "80.0.3987.87"
 }

Firefox:

{
        "name": "firefox "
        , "type": "browser"
        , "browserName": "firefox"
 }

There is a space on the firefox name one?

[jan-molak]

Thanks, and what version of Firefox are you running?

Could you please also add the following section to your protractor.conf.js and let me know what it prints?

    onPrepare: () => {
        return require('protractor').protractor.browser.getCapabilities()
            .then(capabilities => {
                console.log('capabilities',    capabilities);
            });
    },

I think I know what the issue is...

[jan-molak]

Hey @robin-weller, can you check out the 2.1.1 build, please? It should be able to correctly detect details of all the major browsers and platforms.

Scenario view:

Summary view:

As always, thoughts and feedback welcome!

[ctaepper]

seems like I created an issue about the same root cause in #456 I will test the latest version and see if the problem goes away

[jan-molak]

Thanks @ctaepper - yeah, my original solution was a tad bit optimistic. I then ran a test against 14 different browser/platform combinations and recorded my findings in this unit test.

If there's anything I missed, please feel free to extend it!

[robin-weller]

Thanks @jan-molak, 2.1.1 is looking good for browser tagging :)

Something I am seeing a difference in though is the platform tagging.

Chrome:

Browser: Chrome 80.0.3987.87
Platform: Mac OS X

Firefox:

Browser: Firefox 71.0
Platform: Mac 19.3.0 

Currently running Mac OS X 10.15.3 so it looks as though the Firefox run is picking up the Darwin version instead of the OS X version?

[jan-molak]

Hey @robin-weller - Different browsers describe the platform they run on differently, and to a different degree of fidelity. If I was to make the reporting consistent, I'd have to go either with the lowest common denominator or introduce another, higher-level tag (PlatformType?)

So either:

// Chrome:
Browser: Chrome 80.0.3987.87
Platform: Mac

// Firefox:
Browser: Firefox 71.0
Platform: Mac

or:

// Chrome:
Browser: Chrome 80.0.3987.87
Platform: Mac OS X
Platform Type: Mac

// Firefox:
Browser: Firefox 71.0
Platform: Mac 19.3.0 
Platform Type: Mac

Not sure which one is better. Thoughts?

[robin-weller]

I guess it's not going to be perfect either way. If you run one instance of Chrome on Catalina and one on Mohave, you won't see any difference. However, running Firefox would give you a bit more to go on, albeit a Darwin version number.

Having both Platform and Platform Type would provide the most useful information but could end up cluttering the report when different browsers report different information!

[jan-molak]

This functionality has been introduced in @serenity-js/serenity-bdd v2.3.0, which generates both the browser and platform tags based on the information provided by the browsers (and some provide more precise information than others...):

It also generates icons for the recognised browsers and platforms:

---

If Serenity/JS has made your life easier, please consider becoming a sponsor and buying us a coffee every once in a while