New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing assertion is causing test to wait forever #160

Closed
AlexandrosD opened this Issue Jul 2, 2016 · 14 comments

Comments

Projects
None yet
6 participants
@AlexandrosD

AlexandrosD commented Jul 2, 2016

Hello,

I am setting up the test infrastructure for an Angular2 application, with protractor, protractor-cucumber-framework, cucumber-tsflow, and chai/chai-as-promised as the assertion framework.

In the step definition file, Chai and Chai-as-promised are imported as below:

import * as chai from 'chai';
import * as chaiAsPromised from 'chai-as-promised';
var expect = chai.use(chaiAsPromised).expect;
var should = chai.use(chaiAsPromised).should();

Then there is the following step definition:

@then(/^something$/)
  private thenLoginIsSuccessful(callback: cucumber.CallbackStepDefinition): void {
    expect(alertPage.getTitle()).to.eventually.equal('Alert2').and.notify(callback);
    callback();
  };

The alertPage.getTitle() method gets the tilte of a page and returns Promise<string>.
The title is actually "Alert".
When the assertion is the following, expect(alertPage.getTitle()).to.eventually.equal('Alert').and.notify(callback);, the test completes successfully.
However, when the equal part is equal('Alert2'), like in the above example, the test is freezing when it reaches this point, and never completes.

No error appears in the console.

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Jul 2, 2016

Owner

I'm sorry to say this is way too complicated for me to debug. This involves lots of technologies that I've never heard of or used, any of which could be the problem. You're using syntax that doesn't work in any of my browsers or Node.js environments.

We can leave this open in case someone else is able to help, but if you want help from me then you'll have to reduce this to something involving only normal JavaScript, in 10-line file I can run standalone in the browser or node.js.

Owner

domenic commented Jul 2, 2016

I'm sorry to say this is way too complicated for me to debug. This involves lots of technologies that I've never heard of or used, any of which could be the problem. You're using syntax that doesn't work in any of my browsers or Node.js environments.

We can leave this open in case someone else is able to help, but if you want help from me then you'll have to reduce this to something involving only normal JavaScript, in 10-line file I can run standalone in the browser or node.js.

@AlexandrosD

This comment has been minimized.

Show comment
Hide comment
@AlexandrosD

AlexandrosD Jul 2, 2016

Thank you @domenic for responding so quickly.

This is actually Typescript.

I spotted that the 'hanging' issue was caused because I was using both .and.notify(callback); and callback(); calls.

The 'hanging' issue disappeared after removing the callback() call from the step definition, as below:

@then(/^something$/)
  private thenLoginIsSuccessful(callback: cucumber.CallbackStepDefinition): void {
    expect(alertPage.getTitle()).to.eventually.equal('Alert2').and.notify(callback);
  };

Now I see the following error in the results:

Failures:

1) Scenario: The user is fiddling with the alert showcase page - e2e\features\alert\alert.feature:12
   Step: Then title should be 'Alert2' - e2e\features\alert\alert.feature:7
   Step Definition: node_modules\cucumber-tsflow\dist\BindingDecorator.js:143
   Message:
     function timed out after 5000 milliseconds

I would expect an error saying something similar to that title is 'Alert' instead of 'Alert2', the actual error is function timed out after 5000 milliseconds

While now it does not seem (at least to me) to be caused by chai-as-promised, any insight (from you or anyone else) would be appreciated.

AlexandrosD commented Jul 2, 2016

Thank you @domenic for responding so quickly.

This is actually Typescript.

I spotted that the 'hanging' issue was caused because I was using both .and.notify(callback); and callback(); calls.

The 'hanging' issue disappeared after removing the callback() call from the step definition, as below:

@then(/^something$/)
  private thenLoginIsSuccessful(callback: cucumber.CallbackStepDefinition): void {
    expect(alertPage.getTitle()).to.eventually.equal('Alert2').and.notify(callback);
  };

Now I see the following error in the results:

Failures:

1) Scenario: The user is fiddling with the alert showcase page - e2e\features\alert\alert.feature:12
   Step: Then title should be 'Alert2' - e2e\features\alert\alert.feature:7
   Step Definition: node_modules\cucumber-tsflow\dist\BindingDecorator.js:143
   Message:
     function timed out after 5000 milliseconds

I would expect an error saying something similar to that title is 'Alert' instead of 'Alert2', the actual error is function timed out after 5000 milliseconds

While now it does not seem (at least to me) to be caused by chai-as-promised, any insight (from you or anyone else) would be appreciated.

@keithamus

This comment has been minimized.

Show comment
Hide comment
@keithamus

keithamus Jul 2, 2016

Collaborator

an error like function timed out after 5000 milliseconds is typically the test runner (I'm guessing mocha) timing out on the assertion. Presumably alertPage.getTitle() is not resolving after 5s and as such the test fails.

Collaborator

keithamus commented Jul 2, 2016

an error like function timed out after 5000 milliseconds is typically the test runner (I'm guessing mocha) timing out on the assertion. Presumably alertPage.getTitle() is not resolving after 5s and as such the test fails.

@AlexandrosD

This comment has been minimized.

Show comment
Hide comment
@AlexandrosD

AlexandrosD Jul 3, 2016

Thanks @keithamus

Actually I am not using mocha, but cucumber-tsflow, which is from where this error appears to be coming from.

I raised the relevant issue, timjroberts/cucumber-js-tsflow#4, so let's see...

AlexandrosD commented Jul 3, 2016

Thanks @keithamus

Actually I am not using mocha, but cucumber-tsflow, which is from where this error appears to be coming from.

I raised the relevant issue, timjroberts/cucumber-js-tsflow#4, so let's see...

@AlexandrosD AlexandrosD changed the title from Failing assertion is causin test to wait forever to Failing assertion is causing test to wait forever Jul 3, 2016

@jan-molak

This comment has been minimized.

Show comment
Hide comment
@jan-molak

jan-molak Jul 5, 2016

Hey @AlexandrosD,

I'm using a similar tech stack on one of my projects, and so far it seems to me that the problem is not necessarily related to cucumber.js as suggested in timjroberts/cucumber-js-tsflow#4, but more to the Protractor/WebDriver.js combo and how their Control Flow interacts with chai-as-promised.

As per @timjroberts's comment, the following example works fine, the test fails as expected, because so far we're only using ES6 Promises:

@then(/^some then step$/)
public ThenTest(): Promise<any> {
    return chai.expect(Promise.resolve("Hello")).to.eventually.equal("Hello2");
}

When we throw Protractor (or WebDriverJs to be precise) into the mix, the below test times out:

@then(/^some then step$/)
public ThenTest(): Promise<any> {
    return expect(browser.getTitle()).to.eventually.equal('some incorrect title');
}

However, a slightly modified version of the above scenario works perfectly fine:

@then(/^some then step$/)
public ThenTest(): Promise<any> {
    return browser.getTitle().then(title => expect(title).to.equal('some incorrect title'));
}

To my current understanding, the difference between the second and the third scenario comes down to the fact that Protractor relies on WebDriverJs Control Flow to schedule and execute its promises (see the documentation).

Also, it looks to me that if chai-as-promised in the expression:

expect(browser.getTitle()).to.eventually.equal('some incorrect title')

returned the original promise object (generated by browser.getTitle() and already scheduled on the Control Flow), with any assertions chained on top of it, it should all work fine together.

@domenic, @keithamus - I'm keen to hear your thoughts!

Jan

jan-molak commented Jul 5, 2016

Hey @AlexandrosD,

I'm using a similar tech stack on one of my projects, and so far it seems to me that the problem is not necessarily related to cucumber.js as suggested in timjroberts/cucumber-js-tsflow#4, but more to the Protractor/WebDriver.js combo and how their Control Flow interacts with chai-as-promised.

As per @timjroberts's comment, the following example works fine, the test fails as expected, because so far we're only using ES6 Promises:

@then(/^some then step$/)
public ThenTest(): Promise<any> {
    return chai.expect(Promise.resolve("Hello")).to.eventually.equal("Hello2");
}

When we throw Protractor (or WebDriverJs to be precise) into the mix, the below test times out:

@then(/^some then step$/)
public ThenTest(): Promise<any> {
    return expect(browser.getTitle()).to.eventually.equal('some incorrect title');
}

However, a slightly modified version of the above scenario works perfectly fine:

@then(/^some then step$/)
public ThenTest(): Promise<any> {
    return browser.getTitle().then(title => expect(title).to.equal('some incorrect title'));
}

To my current understanding, the difference between the second and the third scenario comes down to the fact that Protractor relies on WebDriverJs Control Flow to schedule and execute its promises (see the documentation).

Also, it looks to me that if chai-as-promised in the expression:

expect(browser.getTitle()).to.eventually.equal('some incorrect title')

returned the original promise object (generated by browser.getTitle() and already scheduled on the Control Flow), with any assertions chained on top of it, it should all work fine together.

@domenic, @keithamus - I'm keen to hear your thoughts!

Jan

@AlexandrosD

This comment has been minimized.

Show comment
Hide comment
@AlexandrosD

AlexandrosD Jul 6, 2016

Well that seems reasonable. Protractor's promises are something different than ES6 promises.
But in any case it looks like a compatibility issue between WebdriverJS and Cucumber - works fine with cucumber@1.0.0 though.

AlexandrosD commented Jul 6, 2016

Well that seems reasonable. Protractor's promises are something different than ES6 promises.
But in any case it looks like a compatibility issue between WebdriverJS and Cucumber - works fine with cucumber@1.0.0 though.

@jan-molak

This comment has been minimized.

Show comment
Hide comment
@jan-molak

jan-molak Jul 6, 2016

What leads me to believe that the issue is not related to WebDriverJs/Cucumber 1.2.0 compatibility is that both libraries work together just fine without Chai, as per the examples earlier.

As long as the ManagedPromise object returned from a WebDriver call is passed intact onto Cucumber when the test step finishes - both tools seem to be perfectly happy together.

If my hypothesis is correct, then:

  • the promise object that the expect(...) function receives is not the same object that it returns (I'm guessing it's probably wrapped into some other promise to support multi-step assertions? That's what confuses WebDriverJs I think)
  • if expect(...) returned the promise it received and chained the assertions on top of it, the problem we've observed would be resolved.

Downgrading to Cucumber@1.0.0 is an option, so is using Chai without Chai as Promised, nevertheless I'm a big fan of the Chai as Promised DSL and would like to use it without workarounds if possible. I'm happy to help with investigating the problem further?

Having said that, I'm not that familiar with the internals of chai-as-promised that's why I was hoping to learn what @domenic and @keithamus think about this?

jan-molak commented Jul 6, 2016

What leads me to believe that the issue is not related to WebDriverJs/Cucumber 1.2.0 compatibility is that both libraries work together just fine without Chai, as per the examples earlier.

As long as the ManagedPromise object returned from a WebDriver call is passed intact onto Cucumber when the test step finishes - both tools seem to be perfectly happy together.

If my hypothesis is correct, then:

  • the promise object that the expect(...) function receives is not the same object that it returns (I'm guessing it's probably wrapped into some other promise to support multi-step assertions? That's what confuses WebDriverJs I think)
  • if expect(...) returned the promise it received and chained the assertions on top of it, the problem we've observed would be resolved.

Downgrading to Cucumber@1.0.0 is an option, so is using Chai without Chai as Promised, nevertheless I'm a big fan of the Chai as Promised DSL and would like to use it without workarounds if possible. I'm happy to help with investigating the problem further?

Having said that, I'm not that familiar with the internals of chai-as-promised that's why I was hoping to learn what @domenic and @keithamus think about this?

@jan-molak

This comment has been minimized.

Show comment
Hide comment
@jan-molak

jan-molak Jul 9, 2016

Update: In order to enable the below syntax in Jasmine tests:

expect(el.getText()).toBe('Hello, World!')

Protractor ships with a WebDriver/Jasmine adapter, which injects Jasmine assertions into WebDriver's Control Flow. Without it, people would need to write this instead:

el.getText().then(function(text) {
  expect(text).toBe('Hello, World!');
});

Bottom line: The problem is caused by incompatibility between standard promises and WebDriver promises executed via the Control Flow.

It could be solved in chai-as-promised (as per my suggestion earlier), but given those recent finding this doesn't seem like the right place to do it. A cleaner approach seems to be to wrap WebDriver promises into standard promises and pass those onto the expect(...) function.

Hope this post helps anyone who stumbles upon this issue in the future,
Jan

jan-molak commented Jul 9, 2016

Update: In order to enable the below syntax in Jasmine tests:

expect(el.getText()).toBe('Hello, World!')

Protractor ships with a WebDriver/Jasmine adapter, which injects Jasmine assertions into WebDriver's Control Flow. Without it, people would need to write this instead:

el.getText().then(function(text) {
  expect(text).toBe('Hello, World!');
});

Bottom line: The problem is caused by incompatibility between standard promises and WebDriver promises executed via the Control Flow.

It could be solved in chai-as-promised (as per my suggestion earlier), but given those recent finding this doesn't seem like the right place to do it. A cleaner approach seems to be to wrap WebDriver promises into standard promises and pass those onto the expect(...) function.

Hope this post helps anyone who stumbles upon this issue in the future,
Jan

@AlexandrosD

This comment has been minimized.

Show comment
Hide comment
@AlexandrosD

AlexandrosD Jul 9, 2016

For sure I would expect that webdriver.promise.Promise would be some day a
standard es6 promise instead.
All of us will need to adopt the standards at the end. That is why they are
called "standards" anyway.
On Jul 10, 2016 02:23, "Jan Molak" notifications@github.com wrote:

Update: In order to enable the below syntax in Jasmine tests:

expect(el.getText()).toBe('Hello, World!')

Protractor ships
https://github.com/angular/protractor/blob/9144494a28dac5a0409de4c5384e933f2d2f8156/docs/webdriver-vs-protractor.md#jasmine-integration
with a WebDriver/Jasmine adapter https://github.com/angular/jasminewd,
which injects
https://github.com/angular/jasminewd/blob/a8ca7e325be67a9228addadb24495585836390c3/index.js#L70
Jasmine assertions into WebDriver's Control Flow. Without it, people would
need to write this instead:

el.getText().then(function(text) {
expect(text).toBe('Hello, World!');
});

Bottom line: The problem is caused by incompatibility between standard
promises and WebDriver promises executed via the Control Flow.

It could be solved in chai-as-promised (as per my suggestion earlier),
but given those recent finding this doesn't seem like the right place to do
it. A cleaner approach seems to be to wrap WebDriver promises into standard
promises and pass those onto the expect(...) function.

Hope this post helps anyone who stumbles upon this issue in the future,
Jan


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#160 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAbExGPvDLW1bH8BmcZIi5P4nhfsswJhks5qUC1tgaJpZM4JDr-6
.

AlexandrosD commented Jul 9, 2016

For sure I would expect that webdriver.promise.Promise would be some day a
standard es6 promise instead.
All of us will need to adopt the standards at the end. That is why they are
called "standards" anyway.
On Jul 10, 2016 02:23, "Jan Molak" notifications@github.com wrote:

Update: In order to enable the below syntax in Jasmine tests:

expect(el.getText()).toBe('Hello, World!')

Protractor ships
https://github.com/angular/protractor/blob/9144494a28dac5a0409de4c5384e933f2d2f8156/docs/webdriver-vs-protractor.md#jasmine-integration
with a WebDriver/Jasmine adapter https://github.com/angular/jasminewd,
which injects
https://github.com/angular/jasminewd/blob/a8ca7e325be67a9228addadb24495585836390c3/index.js#L70
Jasmine assertions into WebDriver's Control Flow. Without it, people would
need to write this instead:

el.getText().then(function(text) {
expect(text).toBe('Hello, World!');
});

Bottom line: The problem is caused by incompatibility between standard
promises and WebDriver promises executed via the Control Flow.

It could be solved in chai-as-promised (as per my suggestion earlier),
but given those recent finding this doesn't seem like the right place to do
it. A cleaner approach seems to be to wrap WebDriver promises into standard
promises and pass those onto the expect(...) function.

Hope this post helps anyone who stumbles upon this issue in the future,
Jan


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#160 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAbExGPvDLW1bH8BmcZIi5P4nhfsswJhks5qUC1tgaJpZM4JDr-6
.

@jan-molak

This comment has been minimized.

Show comment
Hide comment
@jan-molak

jan-molak Sep 12, 2016

@AlexandrosD - I spent some time investigating the differences between the WebDriver Promises and the standard es6 promises, and how those differences affect libraries such as Chai-as-promised. The problem we observed is not related to chai-as-promised.

It turns out that in order to make Cucumber.js, Chai-as-promised and Protractor combo work together in perfect harmony what you need is an adapter, similar to the one for Jasmine I mentioned earlier, or the one that selenium-webdriver provides for mocha.

Since I couldn't find any existing solution, I decided to write such an adapter myself.
It ships together with my new acceptance testing library - Serenity/JS, which you might find interesting as well - here's a brief introduction.

Hope this helps,
Jan

jan-molak commented Sep 12, 2016

@AlexandrosD - I spent some time investigating the differences between the WebDriver Promises and the standard es6 promises, and how those differences affect libraries such as Chai-as-promised. The problem we observed is not related to chai-as-promised.

It turns out that in order to make Cucumber.js, Chai-as-promised and Protractor combo work together in perfect harmony what you need is an adapter, similar to the one for Jasmine I mentioned earlier, or the one that selenium-webdriver provides for mocha.

Since I couldn't find any existing solution, I decided to write such an adapter myself.
It ships together with my new acceptance testing library - Serenity/JS, which you might find interesting as well - here's a brief introduction.

Hope this helps,
Jan

@AlexandrosD

This comment has been minimized.

Show comment
Hide comment
@AlexandrosD

AlexandrosD Sep 14, 2016

Thanks for the feedback @jan

The adapter and your library look promising - I've made a note to try them
the soonest

2016-09-12 15:07 GMT+03:00 Jan Molak notifications@github.com:

@AlexandrosD https://github.com/AlexandrosD - I spent some time
investigating the differences between the WebDriver Promises
http://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/lib/promise.html
and the standard es6 promises, and how those differences affect libraries
such as Chai-as-promised. The problem we observed is not related to
chai-as-promised.

It turns out that in order to make Cucumber.js, Chai-as-promised and
Protractor combo work together in perfect harmony what you need is an
adapter, similar to the one for Jasmine I mentioned earlier
https://github.com/angular/jasminewd, or the one that selenium-webdriver
https://github.com/SeleniumHQ/selenium/tree/eb6c8b218d9311da0df137bc88dd247a02cc7728/javascript/node/selenium-webdriver provides
for mocha
https://github.com/SeleniumHQ/selenium/blob/eb6c8b218d9311da0df137bc88dd247a02cc7728/javascript/node/selenium-webdriver/testing/index.js#L217
.

Since I couldn't find any existing solution, I decided to write such an
adapter myself
https://github.com/jan-molak/serenity-js/blob/cf2d157a4dd30888e68d4ad4934b2c4ad3cf4ef4/src/serenity-cucumber/webdriver_synchroniser.ts.

It ships together with my new acceptance testing library - Serenity/JS
https://github.com/jan-molak/serenity-js, which you might find
interesting as well - here's a brief introduction
https://janmolak.com/introducing-serenity-js-c5565d295702#.etvr9jpv9.

Hope this helps,
Jan


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#160 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAbExLHLRqhKPgKL226JlOKgRAETxMS6ks5qpUCJgaJpZM4JDr-6
.

AlexandrosD commented Sep 14, 2016

Thanks for the feedback @jan

The adapter and your library look promising - I've made a note to try them
the soonest

2016-09-12 15:07 GMT+03:00 Jan Molak notifications@github.com:

@AlexandrosD https://github.com/AlexandrosD - I spent some time
investigating the differences between the WebDriver Promises
http://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/lib/promise.html
and the standard es6 promises, and how those differences affect libraries
such as Chai-as-promised. The problem we observed is not related to
chai-as-promised.

It turns out that in order to make Cucumber.js, Chai-as-promised and
Protractor combo work together in perfect harmony what you need is an
adapter, similar to the one for Jasmine I mentioned earlier
https://github.com/angular/jasminewd, or the one that selenium-webdriver
https://github.com/SeleniumHQ/selenium/tree/eb6c8b218d9311da0df137bc88dd247a02cc7728/javascript/node/selenium-webdriver provides
for mocha
https://github.com/SeleniumHQ/selenium/blob/eb6c8b218d9311da0df137bc88dd247a02cc7728/javascript/node/selenium-webdriver/testing/index.js#L217
.

Since I couldn't find any existing solution, I decided to write such an
adapter myself
https://github.com/jan-molak/serenity-js/blob/cf2d157a4dd30888e68d4ad4934b2c4ad3cf4ef4/src/serenity-cucumber/webdriver_synchroniser.ts.

It ships together with my new acceptance testing library - Serenity/JS
https://github.com/jan-molak/serenity-js, which you might find
interesting as well - here's a brief introduction
https://janmolak.com/introducing-serenity-js-c5565d295702#.etvr9jpv9.

Hope this helps,
Jan


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#160 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAbExLHLRqhKPgKL226JlOKgRAETxMS6ks5qpUCJgaJpZM4JDr-6
.

@spamguy

This comment has been minimized.

Show comment
Hide comment
@spamguy

spamguy Jan 13, 2017

Possibly seeing a similar thing with a more basic setup. Running this using the latest version of Mocha/Karma, plus PhantomJS, plus Bluebird (because my app uses it):

describe('Core Karma functionality', function() {
    it('resolves promises using \'eventually\'', function() {
        return expect(Promise.resolve(123)).to.eventually.equal(123);
    });
});

The result is always a timeout error. Now, take away Bluebird and run this on Chrome (because PhantomJS isn't ES6-ready), and the test doesn't time out.

I was split between noting this here, in #169, or a new ticket, so feel free to redirect this comment.

spamguy commented Jan 13, 2017

Possibly seeing a similar thing with a more basic setup. Running this using the latest version of Mocha/Karma, plus PhantomJS, plus Bluebird (because my app uses it):

describe('Core Karma functionality', function() {
    it('resolves promises using \'eventually\'', function() {
        return expect(Promise.resolve(123)).to.eventually.equal(123);
    });
});

The result is always a timeout error. Now, take away Bluebird and run this on Chrome (because PhantomJS isn't ES6-ready), and the test doesn't time out.

I was split between noting this here, in #169, or a new ticket, so feel free to redirect this comment.

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Jun 11, 2017

Owner

@spamguy that sounds like a Karma issue; I know they like to muck with async stuff to not let it run normally. But I'd suggest opening an issue there.

Since nobody has provided a small repro without lots of third-party technologies, let me close this issue.

Owner

domenic commented Jun 11, 2017

@spamguy that sounds like a Karma issue; I know they like to muck with async stuff to not let it run normally. But I'd suggest opening an issue there.

Since nobody has provided a small repro without lots of third-party technologies, let me close this issue.

@domenic domenic closed this Jun 11, 2017

@balajilinks

This comment has been minimized.

Show comment
Hide comment
@balajilinks

balajilinks Mar 15, 2018

This is not chai issue.. In protractor config, we need to add the following line to fix this problem.
ignoreUncaughtExceptions: true,

balajilinks commented Mar 15, 2018

This is not chai issue.. In protractor config, we need to add the following line to fix this problem.
ignoreUncaughtExceptions: true,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment