Run several instances of codeceptJS #439

Merged
merged 49 commits into from Mar 29, 2017

Conversation

Projects
None yet
2 participants
@APshenkin
Collaborator

APshenkin commented Mar 6, 2017

New feature is most for CI run. Usual, You want to run several suites for several browsers with different window size. Previously for this you have to create several jobs in your CI, or create bash script to run several suites (but they will performed sequentially).
Now there is several options to run suites in parallel with one command.

Now there is run:multiple --multiple option to set what suites to run.

First, create section in your codeceptjs conf like this:

multiple: {
        chrome: {
            smoke: {
                grep: '@smoke28',
                windowSizes: ['maximize','1200x840']
            },
            regression: {
                grep: '@smoke29'
            }
        },
        firefox: {
            smoke: {
                grep: '@smoke28',
                windowSizes: ['maximize','1200x840']
            },
            regression: {
                grep: '@smoke29'
            }
        }
    },

for Each browser you can configure suites. First param in Array is grep parameter to search tests, second one is windowsSizes to launch tests (e.g. if you set two window sizes for smoke, then this suite will launch 2 times, if not set - will set default value from config)

now you can try to run several test cases. Here are examles to run:

codeceptjs run:multiple --multiple 'chrome: smoke; firefox: regression' - will launch smoke suite in chrome and regression in firefox
codeceptjs run:multiple --multiple 'chrome: smoke; regression' - will launch smoke suite in chrome and regression in firefox and chrome
codeceptjs run:multiple --multiple 'all' - will launch all suites
codeceptjs run:multiple --multiple 'smoke, regression' - will launch smoke and regression in firefox and chrome

You can combine launch parameters as you want.

When running multiple you will see suite tag like [ browser_suite_windowSize ] in console for each test (recomended to set steps to false, to provide readable logs)

Also output folder for screenshots will be different for suites: [output_param_form codeceptjs_conf]/browser_suite_windowSize.

Also if you use mochawesome html reporter, the output folder will be also changed

@APshenkin APshenkin changed the title from Run several instances of codeceptJS to Run several instances of codeceptJS [WIP] Mar 6, 2017

@APshenkin APshenkin changed the title from Run several instances of codeceptJS [WIP] to Run several instances of codeceptJS Mar 6, 2017

@DavertMik

This comment has been minimized.

Show comment
Hide comment
@DavertMik

DavertMik Mar 8, 2017

Member

JUST WOW 👍

Member

DavertMik commented Mar 8, 2017

JUST WOW 👍

@DavertMik

DavertMik requested changes Mar 8, 2017 edited

Thanks for the PR. It's absolutely amazing. There are some ideas, I'd like to propose to improve it.

Make more explicit config syntax.

How about:

multiple: {
        chrome: {
            smoke: {
                 grep: '@smoke49', windowSize: ['maximize','1200x840'],
            }               
            regression: { grep: '@smoke48' }
      },
        firefox: {
            smoke: { grep: '@smoke49', windowSize: '1200x840'},
            regression: { grep: '@smoke48' }
      }
    },

If at some point we will introduce groups or tags, it will be easy to add option other than grep to filter out features.

Move run multiple to other command

With this change run becomes bloated with implementation for this feature only. It would be hard to introduce new improvements and fixes into it. So I think you can make a new run-multiple command which extends (I know, you ❤️ this word 🤣 ) the basic run.

lib/command/run.js
+ let suiteConf = config.multiple[browser][suite];
+ let sizes = (suiteConf[1] === undefined) ? ['default'] : suiteConf[1].replace(/ /g, "").split(',')
+ // override browser
+ let override_br = replaceValue(new getConfig(testRoot, configFile), 'browser', (browser === 'ie') ? 'internet explorer' : browser)

This comment has been minimized.

@DavertMik

DavertMik Mar 8, 2017

Member

please give a better name to this variable
overriddenConfig would be ok

@DavertMik

DavertMik Mar 8, 2017

Member

please give a better name to this variable
overriddenConfig would be ok

lib/command/run.js
+
+
+ for (let j = 0; j < suites.length; j++) {
+ for (let l = 0; l < Object.keys(config.multiple).length; l++) {

This comment has been minimized.

@DavertMik

DavertMik Mar 8, 2017

Member

not sure, but can this be written using forEach or for..in which looks nicer to me for languages other than C :)

@DavertMik

DavertMik Mar 8, 2017

Member

not sure, but can this be written using forEach or for..in which looks nicer to me for languages other than C :)

@APshenkin

This comment has been minimized.

Show comment
Hide comment
@APshenkin

APshenkin Mar 13, 2017

Collaborator

Hi @DavertMik !

All changes after review are in the branch. Please check it 🚀

Collaborator

APshenkin commented Mar 13, 2017

Hi @DavertMik !

All changes after review are in the branch. Please check it 🚀

@DavertMik

This comment has been minimized.

Show comment
Hide comment
@DavertMik

DavertMik Mar 29, 2017

Member

Thanks!

Member

DavertMik commented Mar 29, 2017

Thanks!

@DavertMik DavertMik merged commit ace806b into Codeception:master Mar 29, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@DavertMik

This comment has been minimized.

Show comment
Hide comment
@DavertMik

DavertMik Mar 30, 2017

Member

However, I will do some changes; I think the API can be improved.

I'd prefer this syntax:

codeceptjs run:multiple --all                             - will launch all suites
codeceptjs run:multiple smoke regression                  - launch smoke and regression suites
codeceptjs run:multiple smoke:chrome regression:firefox   - will launch smoke suite in chrome and regression in firefox
codeceptjs run:multiple smoke:chrome regression           - will launch smoke suite in chrome and regression in firefox and chrome
  • no additional --multiple tag
  • using params instead of parsing them manually
  • minimal and consistent for different cases

Configuration can be changed accordingly. If the suite is a first, and browser is a second (like it was in your example: codeceptjs run:multiple --multiple 'smoke, regression', then it can be a good idea to adopt config to follow this rule. This sill also allow to make browser config to be more consistent. Here is my proposal:

multiple: {
  smoke: {
    grep: '@smoke28',      
    browsers: [        
      'firefox'
      {browser: 'chrome', windowSize: 'maximize'}, // replace any config values from WebDriverIO helper
      {browser: 'chrome', windowSize: '1200x840'}
    ],
  },
  regression: {
    grep: '@smoke29',
    browsers: ['firefox', 'chrome']
  }
}

This actually is the same config, but it may look nicer. Also browsers option is really general, so run:multiple can be used even with Nightmare, where you can't choose browsers, but you will be able to change the windowSize and other execution options:

  smoke: {
    grep: '@smoke28',      
    browsers: [        
      {windowSize: 'maximize'},
      {windowSize: '200x300'}
    ],
}

I already merged your PR so I will work further, to update the syntax changes and write few basic tests for it.

Member

DavertMik commented Mar 30, 2017

However, I will do some changes; I think the API can be improved.

I'd prefer this syntax:

codeceptjs run:multiple --all                             - will launch all suites
codeceptjs run:multiple smoke regression                  - launch smoke and regression suites
codeceptjs run:multiple smoke:chrome regression:firefox   - will launch smoke suite in chrome and regression in firefox
codeceptjs run:multiple smoke:chrome regression           - will launch smoke suite in chrome and regression in firefox and chrome
  • no additional --multiple tag
  • using params instead of parsing them manually
  • minimal and consistent for different cases

Configuration can be changed accordingly. If the suite is a first, and browser is a second (like it was in your example: codeceptjs run:multiple --multiple 'smoke, regression', then it can be a good idea to adopt config to follow this rule. This sill also allow to make browser config to be more consistent. Here is my proposal:

multiple: {
  smoke: {
    grep: '@smoke28',      
    browsers: [        
      'firefox'
      {browser: 'chrome', windowSize: 'maximize'}, // replace any config values from WebDriverIO helper
      {browser: 'chrome', windowSize: '1200x840'}
    ],
  },
  regression: {
    grep: '@smoke29',
    browsers: ['firefox', 'chrome']
  }
}

This actually is the same config, but it may look nicer. Also browsers option is really general, so run:multiple can be used even with Nightmare, where you can't choose browsers, but you will be able to change the windowSize and other execution options:

  smoke: {
    grep: '@smoke28',      
    browsers: [        
      {windowSize: 'maximize'},
      {windowSize: '200x300'}
    ],
}

I already merged your PR so I will work further, to update the syntax changes and write few basic tests for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment