Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is there a way to set context tags for multi-browser testing? #455

Closed
robin-weller opened this issue Feb 6, 2020 · 16 comments
Closed

Is there a way to set context tags for multi-browser testing? #455

robin-weller opened this issue Feb 6, 2020 · 16 comments

Comments

@robin-weller
Copy link

@robin-weller robin-weller commented Feb 6, 2020

Hi Jan,

With the Java Serenity version, you can provide a context for each browser so that the report shows which browser the test has run on. I have found an old example on John Smart's blog and it still works with the latest version.

Is this something that can easily be done with Serenity/JS?

I am able to run the tests in multiple browsers but there is no distinction in the report to show which browser the test run on.

@abhinaba-ghosh
Copy link

@abhinaba-ghosh abhinaba-ghosh commented Feb 6, 2020

@robin-weller
Actually serenity JS report does show the browser information the nicest way possible. I am working on serenity 1.x version and able to get below information in the report:

image

Not only that, for multi-browser tests, all the browsers will be listed down here.

Most important thing is, before each test case, it shows the icon in which browser tests are executed.

image

Can you share your protractor.conf and the report you are getting. It will ravel a broader picture.

@robin-weller
Copy link
Author

@robin-weller robin-weller commented Feb 6, 2020

That is what I see using the Java version. I am using v2 though.

protractor.conf.js:

const
    { ConsoleReporter } = require('@serenity-js/console-reporter'),
    { ArtifactArchiver } = require('@serenity-js/core'),
    { SerenityBDDReporter } = require('@serenity-js/serenity-bdd'),
    { Photographer, TakePhotosOfInteractions } = require('@serenity-js/protractor');

exports.config = {
    chromeDriver: require('chromedriver/lib/chromedriver').path,
    geckoDriver: require('geckodriver/lib/geckodriver').path,
    SELENIUM_PROMISE_MANAGER: false,

    directConnect: true,

    allScriptsTimeout: 11000,

    specs: [ './features/*.feature', ],

    framework: 'custom',
    frameworkPath: require.resolve('@serenity-js/protractor/adapter'),

    onPrepare: () => {
        require('ts-node').register({
            project: './tsconfig.json'
        });
        browser.waitForAngularEnabled(false);
    },

    serenity: {
        crew: [
            ArtifactArchiver.storingArtifactsAt('./target/site/serenity'),
            Photographer.whoWill(TakePhotosOfInteractions),
            new SerenityBDDReporter(),
            ConsoleReporter.forDarkTerminals(),
        ],
        runner: 'cucumber'
    },

    cucumberOpts: {
        require: [
            'features/steps/**/*.ts',
            'features/support/setup.ts',
        ],
        'require-module': ['ts-node/register'],
    },

    multiCapabilities: [
        {
            browserName: "chrome",
            chromeOptions: {
                args: [
                    '--disable-infobars',
                    '--no-sandbox',
                    '--disable-gpu',
                    '--window-size=1024x768',
                    // '--headless',
                ],
            },
        },
        {
            browserName: "firefox",
        },
    ],
};

Report:
Screenshot 2020-02-06 at 13 57 46

A Java version:
Screenshot 2020-02-06 at 14 01 20

Edit: In the java version, I set -Dcontext=browserName but not sure how I would set this?

@abhinaba-ghosh
Copy link

@abhinaba-ghosh abhinaba-ghosh commented Feb 6, 2020

@jan-molak Is it related to old serenity CLI jar?

@jan-molak
Copy link
Member

@jan-molak jan-molak commented Feb 7, 2020

Hi @robin-weller, it's great to hear from you again!

@abhinaba1080 - your instincts are right, it is partially related to the old Serenity CLI jar. The upcoming 2.1.10 release of the jar will come with serenity-bdd/serenity-core#1860, which enables both browser and OS icons to be rendered for every test scenario.

I was waiting for that jar to be published to add both the browser/os tagging as well as the icons (#132) all in one go. But, having thought about it some more, I decided to implement the tagging first and the icons whenever the jar arrives.

Enjoy Serenity/JS v 2.1.0 with OS and browser tags

image

Support for icons is coming soon ;-)


You can now sponsor my work on Serenity/JS on Github :octocat:

@robin-weller
Copy link
Author

@robin-weller robin-weller commented Feb 7, 2020

Thanks @jan-molak - good to hear from you too!

I am seeing the OS and browser tags in the report, however, there is an issue generating the report from firefox runs. so it is only showing chrome.

I see the following error when it is generating reports from firefox runs:
[main] WARN n.t.c.r.json.JSONTestOutcomeReporter - This file was not a valid JSON Serenity test report: scenario-user-can-log-in-ec36ba7706.json

I am seeing the chrome icon though!

@jan-molak
Copy link
Member

@jan-molak jan-molak commented Feb 7, 2020

Cool, so we're making progress :-) Would you mind posting the offending json file?

@robin-weller
Copy link
Author

@robin-weller robin-weller commented Feb 7, 2020

Of course:

{ 
   "name":"User can log in",
   "title":"User can log in",
   "id":"demo-journeys;user-can-log-in",
   "manual":false,
   "testSteps":[ 
      { 
         "number":1,
         "description":"Given Arya is on the <Obfuscated Website> homepage",
         "startTime":1581086694376,
         "children":[ 
            { 
               "number":2,
               "description":"Arya navigates to '<Obfuscated Website>'",
               "startTime":1581086694379,
               "children":[ 

               ],
               "reportData":[ 

               ],
               "screenshots":[ 
                  { 
                     "screenshot":"photo-firefox-arya-navigates-to-<Obfuscated Website>-cf8036b9c1.png"
                  }
               ],
               "result":"SUCCESS",
               "duration":905
            },
            { 
               "number":3,
               "description":"Arya ensures that the title of the current page does start with '<Obfuscated Website>'",
               "startTime":1581086695287,
               "children":[ 

               ],
               "reportData":[ 

               ],
               "screenshots":[ 
                  { 
                     "screenshot":"photo-firefox-arya-ensures-that-the-title-of-the-current-page-does-start-with-<Obfuscated Website>-b2144a1f3b.png"
                  }
               ],
               "result":"SUCCESS",
               "duration":5
            }
         ],
         "reportData":[ 

         ],
         "screenshots":[ 

         ],
         "result":"SUCCESS",
         "duration":919
      },
      { 
         "number":4,
         "description":"When she logs in",
         "startTime":1581086695296,
         "children":[ 
            { 
               "number":5,
               "description":"Arya logs in to <Obfuscated Website>",
               "startTime":1581086695297,
               "children":[ 
                  { 
                     "number":6,
                     "description":"Arya clicks on the the user icon",
                     "startTime":1581086695297,
                     "children":[ 

                     ],
                     "reportData":[ 

                     ],
                     "screenshots":[ 
                        { 
                           "screenshot":"photo-firefox-arya-clicks-on-the-the-user-icon-dad1f72b8e.png"
                        }
                     ],
                     "result":"SUCCESS",
                     "duration":2581
                  },
                  { 
                     "number":7,
                     "description":"Arya enters '<Obfuscated Email>' into the the email address field",
                     "startTime":1581086697879,
                     "children":[ 

                     ],
                     "reportData":[ 

                     ],
                     "screenshots":[ 
                        { 
                           "screenshot":"photo-firefox-arya-enters-<Obfuscated Email>-into-the-the-email-address-field-1bd27fbedd.png"
                        }
                     ],
                     "result":"SUCCESS",
                     "duration":412
                  },
                  { 
                     "number":8,
                     "description":"Arya enters '<Obfuscated Password>' into the the password field",
                     "startTime":1581086698292,
                     "children":[ 

                     ],
                     "reportData":[ 

                     ],
                     "screenshots":[ 
                        { 
                           "screenshot":"photo-firefox-arya-enters-<Obfuscated Password>-into-the-the-password-field-3673e82b69.png"
                        }
                     ],
                     "result":"SUCCESS",
                     "duration":182
                  },
                  { 
                     "number":9,
                     "description":"Arya clicks on the the sign in button",
                     "startTime":1581086698476,
                     "children":[ 

                     ],
                     "reportData":[ 

                     ],
                     "screenshots":[ 
                        { 
                           "screenshot":"photo-firefox-arya-clicks-on-the-the-sign-in-button-68499d3cfd.png"
                        }
                     ],
                     "result":"SUCCESS",
                     "duration":513
                  }
               ],
               "reportData":[ 

               ],
               "screenshots":[ 

               ],
               "result":"SUCCESS",
               "duration":3693
            }
         ],
         "reportData":[ 

         ],
         "screenshots":[ 

         ],
         "result":"SUCCESS",
         "duration":3695
      },
      { 
         "number":10,
         "description":"Then she should see that she is logged in",
         "startTime":1581086698991,
         "children":[ 

         ],
         "reportData":[ 

         ],
         "screenshots":[ 

         ],
         "result":"PENDING",
         "duration":1
      }
   ],
   "userStory":{ 
      "id":"demo-journeys",
      "storyName":"Demo Journeys",
      "path":"features/demo.feature",
      "type":"feature"
   },
   "startTime":1581086694369,
   "testSource":"Cucumber",
   "tags":[ 
      { 
         "name":"Demo Journeys",
         "type":"feature"
      },
      { 
         "name":"firefox ",
         "type":"browser",
         "browserName":"firefox"
      },
      { 
         "type":"platform"
      }
   ],
   "featureTag":{ 
      "name":"Demo Journeys",
      "type":"feature"
   },
   "context":"firefox",
   "driver":"firefox",
   "duration":4811,
   "result":"PENDING"
}

@robin-weller
Copy link
Author

@robin-weller robin-weller commented Feb 7, 2020

The difference I am seeing between the Chome and Firefox JSON files is there is no recorded browserVersion for Firefox (which is probably fine?!) and there is no name defined with "type": "platform"?

One final thing:
Chrome:

{
        "name": "chrome 80.0.3987.87"
        , "type": "browser"
        , "browserName": "chrome"
        , "browserVersion": "80.0.3987.87"
 }

Firefox:

{
        "name": "firefox "
        , "type": "browser"
        , "browserName": "firefox"
 }

There is a space on the firefox name one?

@jan-molak
Copy link
Member

@jan-molak jan-molak commented Feb 7, 2020

Thanks, and what version of Firefox are you running?

Could you please also add the following section to your protractor.conf.js and let me know what it prints?

    onPrepare: () => {
        return require('protractor').protractor.browser.getCapabilities()
            .then(capabilities => {
                console.log('capabilities',    capabilities);
            });
    },

I think I know what the issue is...

jan-molak added a commit that referenced this issue Feb 8, 2020
@jan-molak
Copy link
Member

@jan-molak jan-molak commented Feb 8, 2020

Hey @robin-weller, can you check out the 2.1.1 build, please? It should be able to correctly detect details of all the major browsers and platforms.

Scenario view:
image

Summary view:
image

As always, thoughts and feedback welcome!

@ctaepper
Copy link

@ctaepper ctaepper commented Feb 8, 2020

seems like I created an issue about the same root cause in #456 I will test the latest version and see if the problem goes away

@jan-molak
Copy link
Member

@jan-molak jan-molak commented Feb 8, 2020

Thanks @ctaepper - yeah, my original solution was a tad bit optimistic. I then ran a test against 14 different browser/platform combinations and recorded my findings in this unit test.

If there's anything I missed, please feel free to extend it!

@robin-weller
Copy link
Author

@robin-weller robin-weller commented Feb 10, 2020

Thanks @jan-molak, 2.1.1 is looking good for browser tagging :)

Something I am seeing a difference in though is the platform tagging.

Chrome:

Browser: Chrome 80.0.3987.87
Platform: Mac OS X

Firefox:

Browser: Firefox 71.0
Platform: Mac 19.3.0 

Currently running Mac OS X 10.15.3 so it looks as though the Firefox run is picking up the Darwin version instead of the OS X version?

@jan-molak
Copy link
Member

@jan-molak jan-molak commented Feb 10, 2020

Hey @robin-weller - Different browsers describe the platform they run on differently, and to a different degree of fidelity. If I was to make the reporting consistent, I'd have to go either with the lowest common denominator or introduce another, higher-level tag (PlatformType?)

So either:

// Chrome:
Browser: Chrome 80.0.3987.87
Platform: Mac

// Firefox:
Browser: Firefox 71.0
Platform: Mac

or:

// Chrome:
Browser: Chrome 80.0.3987.87
Platform: Mac OS X
Platform Type: Mac

// Firefox:
Browser: Firefox 71.0
Platform: Mac 19.3.0 
Platform Type: Mac

Not sure which one is better. Thoughts?

@robin-weller
Copy link
Author

@robin-weller robin-weller commented Feb 10, 2020

I guess it's not going to be perfect either way. If you run one instance of Chrome on Catalina and one on Mohave, you won't see any difference. However, running Firefox would give you a bit more to go on, albeit a Darwin version number.

Having both Platform and Platform Type would provide the most useful information but could end up cluttering the report when different browsers report different information!

@jan-molak jan-molak added this to To do in Serenity/JS Board Mar 10, 2020
Serenity/JS Board automation moved this from To do to Done Mar 15, 2020
@jan-molak
Copy link
Member

@jan-molak jan-molak commented Mar 15, 2020

This functionality has been introduced in @serenity-js/serenity-bdd v2.3.0, which generates both the browser and platform tags based on the information provided by the browsers (and some provide more precise information than others...):

image

It also generates icons for the recognised browsers and platforms:

image


If Serenity/JS has made your life easier, please consider becoming a sponsor and buying us a coffee every once in a while :octocat:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

No branches or pull requests

4 participants