feat(config): Add shardTestSuites config option

[BernardoSilva]

Currently the option shardTestFiles makes the tests run in parallel but by file, if we have tests that have some dependency on the previous one then this approach does not work.

shardTestSuites option will allow to run tests in different instances but grouped by suite.

The main goal of this feature is to reduce the time it takes to run tests on big applications that already have tests grouped by suites.

I am still testing this approach, but would be nice to have more feedback.

There is an example of how I am using this feature.

example to run your suites in two instances:

maxSessions: 2,
  multiCapabilities: [{
    'browserName': 'chrome',
    shardTestSuites: true,
    maxInstances: 2
  }],

[gssatcc]

Nice feature. I am also looking for it.

[NickTomlin]

Can we remove this space? It keeps things in-line with the rest of the project.

[NickTomlin]

This is really interesting. Personally the need for having specs depend on one another feels like a smell, but I do understand need to be pragmatic about setting up state in large applications.

I made some minor stylistic notes but i'll defer judgement to people who know/care more about this feature than I.

To any of the maintainers, please let me know if my comments/feedback is not wanted; my desire is to be helpful, but I understand that it might be distracting or unnecessary :)

[sjelin]

Won't think cause scripts/test.js to fail?

[BernardoSilva]

@NickTomlin This is really handy for example for CRUD operations.
You may have a spec just for creating and testing all the exceptions and validations and then have another spec that test the Read view where you have to list and expect that previous inserted item to be listed to test search and edit on that same item.
Also you may have another spec to test creating but with another user type that my have more options than the previous user, it's good to be able to have more specs separated by context.

@sjelin Added test to ensure that it's running on same suite otherwise it would fail. :)

[BernardoSilva]

@juliemr Thanks for your feedback @ AngularConnect in London.
Can you give some feedback on this feature/approach? :)

[sjelin]

Thanks for the test updates! But you still need to add a test making sure that different suites are being run independently of each other (i.e. that your feature works).

[sjelin]

Please document this in docs/referenceConf.js.

[hankduan]

Can you provide an example where you must have tests that depend on each other? It is usually a bad practice when this happens, and I don't think we should encourage this by adding a feature to support interfile dependency in the core library. However, you may prove me wrong with a good example.

[BernardoSilva]

@hankduan the example I have is from the company where I am currently working where the tests are taking in average 45min to run each build on a large AWS instance.

Some context:

• We have our application separated by modules.
• As good practice we try to separate the spec files by context/page
• In average we try to test one PageObject on each spec file, so we end up with dependencies between files but is much easier to maintain and extend.

I don't want to get into specific details, but for example:

test/e2e/campaign
    - TestCreateItemFeature.js
    - TestListItemFeature.js
    - TestUpdateItemFeature.js
    - TestCreateInstanceOfItemFeature.js
    - TestEditInstanceOfItemFeature.js
    - TestPublishInstanceOfItemFeature.js

In this example:

• Expect to be able to create an item.
• Then expect that item to be available to test features of list, edit, filter and sort.
• Then we expect to be able to create an instance of that item
• Then we expect that instance to be available to test features of list, edit, filter, sort, publish.

It gets a bit more complicated, but I think this is a good example.

We also did some effort to enable to run each suite of tests (modules) independently, this would allow us to run them in parallel.

I've noticed you already support to run tests in x number of browser instances.
I thought this can be a native feature that Protractor could offer to run suites instead of running just individual test spec files.
This is a big feature for large projects and even smaller ones.

@sjelin about the tests:

I've created a really simple example where I navigate into one page on first spec file
And in the next spec file I am just checking if I am on that page
This prove that if the tests where being sharded by files each one would run on a separate browser instance and the second spec file would fail.

Ps: sorry for my delay on my reply.

Let me know if you need anything else.

[hankduan]

I'll use your item tests as an example:

test/e2e/campaign
    - TestCreateItemFeature.js
    - TestListItemFeature.js
    - TestUpdateItemFeature.js

I would highly suggest reorganizing the tests in one of the following ways:

TestCreateItemFeature.js:

describe('create item', function() {
  it('creation test 1', testFunction1())
  it('creation test 2', testFunction2())
  it('creation test 3', testFunction3())
});

TestListItemFeature.js:

describe('list item', function() {
  beforeEach(function() {
    // create your items;
  });
  it('list test 1', testFunction1())
  it('list test 2', testFunction2())
  afterEach(function() {
    // delete your items;
  });
});

TestUpdateItemFeature.js:

describe('update item', function() {
  beforeEach(function() {
    // create your items;
  });
  it('update test 1', testFunction1())
  it('update test 2', testFunction2())
  afterEach(function() {
    // delete your items;
  });
});

---

So yes, this requires you to create and delete your items multiple times, but you will definitely run into issues if your tests start depending on each other. e.g.

• once you set up your dependency (A->B->C), someone down the line could decide...C requires field 'x.foo' to be set, let's change that in 'A'; or B has a side effect that modifies field 'x.bar', so in C I should assert against the new value of 'bar' etc, etc. Then later, your test C is impossible to read since it's require a state for items which was set up across multiples files as side effects. It's a huge rabbit hole to go down.
• every time something is created, there's a purpose. i.e. you wouldn't have 100 items in your page when your test is to update one of the items. It makes debugging harder. Of course, you could decide to have a test where you're trying to select 1 item to update out of 100, but in this case your set up is intentional
• and as a minor point, Jasmine/Protractor does not guarantee your test files to be run in any particular order, although it is in order at the moment

---

Lastly, by no means do I intend for you to have no dependency on different files. My point is for you to not have dependencies on different tests.

back to the example above, I could have a CreateItemPage.js page object.

var createItemPage = {
  goToPage: function () {
    browser.get('...')
  }
  createItem: function (...params) {
    $(...).sendKeys(param1);
    $(...).sendKeys(param2);
    $(...).sendKeys(param3);
    $(...).click();
  }
}

TestCreateItemFeature.js:

describe('create item', function() {
  it('creation test 1', function() {
    createItemPage.createItem('foo', 'bar', 34, 454);
    // assert...
  })
  it('creation test 2', testFunction2())
  it('creation test 3', testFunction3())
});

TestListItemFeature.js:

describe('list item', function() {
  beforeEach(function() {
    // create your items;
    createItemPage.createItem('item1', 'bar', 34, 454);
    createItemPage.createItem('item2', 'bar', 34, 34);
  });
  it('list test 1', testFunction1())
  it('list test 2', testFunction2())
  afterEach(function() {
    // delete your items;
  });
});

TestUpdateItemFeature.js:

describe('update item', function() {
  beforeEach(function() {
    // create your items;
    createItemPage.createItem('item1', 'bar', 34, 454);
    createItemPage.createItem('item2', 'bar', 34, 34);
  });
  it('update test 1', testFunction1())
  it('update test 2', testFunction2())
  afterEach(function() {
    // delete your items;
  });
});

[sjelin]

@BernardoSilva You've got a great test for "Don't shard when I don't tell you to." But you also need a test for "Do shard when do I tell you to."

[osavchenko]

Good feature 👍

[ArturKwiatkowski]

Hello, any updates on this?

@hankduan I also have a example why this would be great to have:

I'm working on E2E's for booking system where the user can actually buy tickets for rail and it's a complicated process because it's divided into steps where each step can be only accessed after previous one has been completed. To add a cherry on the top - on some steps we're calling external services that return data required for that step to be completed so using the shardFiles option is a no-no for me because it's simply not doable and this is why I hope this feature will be accepted. Other thing why it's not doable for me via shardFiles is when you start to book a ticket you get something like temporary reservation which will allow you to complete the process in certain amount of time, so I can't bypass that in any way and I'm forced to run tests in particular sequence.

[juliemr]

I'm cleaning up stale PRs and features, and closing this one. Please open a new PR/issue if you feel it's still valid. Thanks!

[mayankpurohit17]

@juliemr Please suggest when 'shardTestSuites' can be used. What changes will be required in spec file since I want to execute individual it blocks in parallel? Thanks in advance!