Failed assertion with chai-as-promised is not working

[mingfang]

What are you trying to achieve?

I'm trying to use chai-as-promised for assertions.
It works when the assertion passes but it does not work when the assertion fails.

What do you get instead?

I get this error

(node:40795) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): AssertionError: expected { Object (_readableState, readable, ...) } to have property 'foo'
(node:40795) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

and the test is marked as passed when though it should fail.

Provide console output if related. Use --verbose mode for more details.

./node_modules/codeceptjs/bin/codecept.js run  --verbose
CodeceptJS v1.1.2

   [1] Starting recording promises
   Emitted | suite.before ([object Object])
   [1] Queued | hook Nightmare._beforeSuite()
   [1] Queued | hook Mochawesome._beforeSuite()
   [1] Queued | hook REST._beforeSuite()
   Emitted | test.before
   [1] Queued | hook Nightmare._before()
   [1] Queued | hook Mochawesome._before()
   [1] Queued | hook REST._before()
   Emitted | test.start ([object Object])
   Emitted | step.before (I send get request "http://google.com")
   [1] Queued | hook Nightmare._beforeStep()
   [1] Queued | hook Mochawesome._beforeStep()
   [1] Queued | hook REST._beforeStep()
   [1] Queued | sendGetRequest: "http://google.com"
   Emitted | step.after (I send get request "http://google.com")
   [1] Queued | hook Nightmare._afterStep()
   [1] Queued | hook Mochawesome._afterStep()
   [1] Queued | hook REST._afterStep()
   [1] Queued | step passed
   [1] Queued | return result
   [1] Queued | fire test.passed
   [1] Queued | finish test
   Emitted | step.start (I send get request "http://google.com")
 • I send get request "http://google.com"
   Emitted | step.passed (I send get request "http://google.com")
   Step finished in 0.06 sec
   Emitted | test.passed ([object Object])
 ✓ OK in 103ms

(node:40934) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): AssertionError: expected { Object (_readableState, readable, ...) } to have property 'foo'
(node:40934) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
   Emitted | test.after
   [1] Queued | hook Nightmare._after()
   [1] Queued | hook Mochawesome._after()
   [1] Queued | hook REST._after()
   Emitted | suite.after ([object Object])
   [1] Queued | hook Nightmare._afterSuite()
   [1] Queued | hook Mochawesome._afterSuite()
   [1] Queued | hook REST._afterSuite()

  OK  | 1 passed   // 1s
   Emitted | global.result ([object Object])
   [1] Queued | hook Nightmare._finishTest()
   [1] Queued | hook Mochawesome._finishTest()
   [1] Queued | hook REST._finishTest()

Provide test source code if related

Scenario('test chai', async(I) => {
  const res =  I.sendGetRequest("http://google.com")
  const chai = require('chai')
  chai.use(require('chai-as-promised'))
  const expect = chai.expect
  expect(res)
    // .to.eventually.have.property("body") //this works
    .to.eventually.have.property("foo") //this does not work
});

Details

• CodeceptJS version: 1.1.2
• NodeJS Version: 8.9.4
• Operating System: OSX

[reubenmiller]

You don't need to use the chai-as-promised as you are using async functions. This means you can write your scenario as:

Scenario('test chai', async(I) => {
  const res =  await I.sendGetRequest("http://google.com");
  const chai = require('chai');
  const expect = chai.expect;
  expect(res).to.have.property("body");
});

[mingfang]

@reubenmiller Your example works. But if I change it to this

Scenario('test chai', async(I) => {
  const res =  await I.sendGetRequest("http://google.com");
  const chai = require('chai');
  const expect = chai.expect;
  expect(res).to.have.property("foo"); //force false assertion
});

Then the test hangs for a 30s timeout. Here is the log.

codecept.js run log_test.js --verbose
CodeceptJS v1.1.2
Using test root "/"

HAR --
   [1] Starting recording promises
   Emitted | suite.before ([object Object])
   [1] Queued | hook Nightmare._beforeSuite()
   [1] Queued | hook Mochawesome._beforeSuite()
   [1] Queued | hook REST._beforeSuite()
 test chai
   Emitted | test.before
   [1] Queued | hook Nightmare._before()
   [1] Queued | hook Mochawesome._before()
   [1] Queued | hook REST._before()
   Emitted | test.start ([object Object])
   Emitted | step.before (I send get request "http://google.com")
   [1] Queued | hook Nightmare._beforeStep()
   [1] Queued | hook Mochawesome._beforeStep()
   [1] Queued | hook REST._beforeStep()
   [1] Queued | sendGetRequest: "http://google.com"
   Emitted | step.after (I send get request "http://google.com")
   [1] Queued | hook Nightmare._afterStep()
   [1] Queued | hook Mochawesome._afterStep()
   [1] Queued | hook REST._afterStep()
   [1] Queued | step passed
   [1] Queued | return result
   Emitted | step.start (I send get request "http://google.com")
 • I send get request "http://google.com"
   Emitted | step.passed (I send get request "http://google.com")
   Step finished in 0.062 sec
   [1] Starting <teardown> session
   Emitted | test.failed ([object Object])
   [1] <teardown> Queued | hook Nightmare._failed()
   [1] <teardown> Queued | hook Mochawesome._failed()
   [1] <teardown> Queued | hook REST._failed()
   [1] <teardown> Queued | () => done(err)
 > Screenshot is saving to /Users/mfang/consensys/balanc3-terraform/tests/output/test_chai_undefined.failed.png
   [1] <teardown> Error | Error: Evaluation timed out after 30000msec.  Are you calling done() or resolving your promises?
Error: Evaluation timed out after 30000msec.  Are you calling done() or resolving your promises?
 ✖ FAILED in 30131ms

   Emitted | test.after
   [1] <teardown> Queued | hook Nightmare._after()
   [1] <teardown> Queued | hook Mochawesome._after()
   [1] <teardown> Queued | hook REST._after()
   Emitted | suite.after ([object Object])
   [1] <teardown> Queued | hook Nightmare._afterSuite()
   [1] <teardown> Queued | hook Mochawesome._afterSuite()
   [1] <teardown> Queued | hook REST._afterSuite()

-- FAILURES:

  1) HAR
       test chai:
     expected { Object (_readableState, readable, ...) } to have property 'foo'
  ssertionError: expected { Object (_readableState, readable, ...) } to have property 'foo'
      at Test.Scenario.only (log_test.js:21:23)
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:188:7)


  FAIL  | 0 passed, 1 failed   // 31s
   Emitted | global.result ([object Object])
   [1] <teardown> Queued | hook Nightmare._finishTest()
   [1] <teardown> Queued | hook Mochawesome._finishTest()
   [1] <teardown> Queued | hook REST._finishTest()

[reubenmiller]

The return properties are probably stored under the body of the response (i.e. res.body.foo not res.foo). So just access the .body response and not the response directly.

i.e.

Scenario('test chai', async(I) => {
  const res =  await I.sendGetRequest("http://google.com");
  const chai = require('chai');
  const expect = chai.expect;
  console.log('res:', res);
  console.log('res body:', res.body);
  expect(res.body).to.have.property("foo");
});

[mingfang]

I just ran your test and I'm getting the 30s timeout.
In fact, any failed assertion has this problem.
e.g. this test causes a timeout

Scenario('test chai', async(I) => {
  const chai = require('chai');
  const expect = chai.expect;
  expect({foo: "bar"}).to.have.property("foofoo");
});

I'm trying to figure out if others have this problem too or if it's just my environment.

[reubenmiller]

I just ran your last scenario in a simple setup, and it works fine. I checked both passing and failing scenarios.

const chai = require('chai');
const expect = chai.expect;

Feature('REST Test Example');

Scenario('rest test chai should pass', async(I) => {
  expect({foo: "bar"}).to.have.property("foo");
});

Scenario('rest test chai should fail', async(I) => {
    expect({foo: "bar"}).to.have.property("foofoo");
});

This is the output I received.

> codeceptjs run --grep 'rest' --reporter mocha-multi

CodeceptJS v1.1.2
Using test root "/Users/reubenmiller/codeceptjs-starter"

REST Test Example --
 rest test chai should pass
 ✓ OK in 6ms

 rest test chai should fail
 ✖ FAILED in 1119ms


-- FAILURES:

  1) REST Test Example
       rest test chai should fail:
     expected { foo: 'bar' } to have property 'foofoo'

  Run with --verbose flag to see NodeJS stacktrace


  FAIL  | 1 passed, 1 failed   // 6s

So it might be something in your setup. Are you using the teardown hook in the config file? Or hooks in any custom helpers?

[reubenmiller]

I've updated my example codeceptjs project with some rest tests if you want to have a look at the full config.

https://github.com/reubenmiller/CodeceptJS-starter

Then to run the rest tests, execute the following

npm run test:rest

[mingfang]

I started to strip everything out of my codecept.json file to make it look like this

{
  "output": "./output",
  "helpers": {
    "Nightmare": {
    }
  },
  "bootstrap": false,
  "teardown": null,
  "hooks": [],
  "tests": "./*_test.js",
  "timeout": 10000,
  "name": "test"
}

It turns out the problem is with the Nightmare helper.
If I remove it from codecept.json then everything works as expected.
Can you please try adding the Nightmare helper to see if the problem happens on our side too?

[reubenmiller]

Yeah I get the same error when switching to the Nightmare helper in my codeceptjs config.

So looks like something is wrong when a test fails when using the Nightmare helper.

[reubenmiller]

I would recommend switching to Puppeteer instead.

[reubenmiller]

I revisited this and played around with the test case, and have something that works. It looks like that when using the Nightmare Helper, you need to navigate to a page first before executing your assertion. I'm not 100% sure why, but at least the work around should be acceptable (but let me know).

Scenario('Test should fail using assert @rest', (I) => {
  I.amOnPage('/'); // This is needed (it might kick off the promise chain
  assert.equal(true, false);
});

and here is the Helper config with the Nighmare configuration.

helpers: {
    Nightmare: {
      url: 'https://google.com',
      windowSize: '600x400',
      show: true,
    },
  },

I hadn't seen this problem before because all of my tests always start out with I.amOnPage('/') (or something similar.

Though, I noticed that in your original test case that you are using I.sendGetRequest, unless you have created this function yourself, then your test scenario is only supported with the REST Helper, and you should not have the REST helper and the Nightmare Helper configured in the same config file.

[dmydry]

Hello guys! I have been stuck with this problem and just realized what's going on.
First of all, please check the next code:

Scenario('Async', async I => {
  assert.equal(false, true);
});

Scenario('Sync', I => {
  assert.equal(false, true);
});

The first one will be passed, but we will se a console message:

Async
 ✓ OK in 3ms

(node:77727) UnhandledPromiseRejectionWarning: AssertionError [ERR_ASSERTION]: false == true

The second one will be failed:

Sync
 ✖ FAILED in 635ms

-- FAILURES:

  1) Test
       Sync:

      false == true
      + expected - actual

      -false
      +true

The console message above is trying to help us and push us to handle a promise rejection. The most simple and common way to do this is to to use try...catch.
My problem was to find a way to reject a test inside a catch. So ended up with the next code:

import { recorder } from 'codeceptjs';

Scenario('Async', async I => {
  try {
    assert.equal(false, true);
  } catch (e) {
    recorder.throw(e);
  }
});

Scenario('Sync', I => {
  assert.equal(false, true);
});

In case if anyone else will be looking for answers.

CodeceptJS Async Await handle promise rejection
CodeceptJS Async assert
CodeceptJS Manually stop test

[reubenmiller]

I think @APshenkin may have picked this up recently in PR #933. The PR has been merged but it is not in a new release yet.

But that is just a guess.

[reubenmiller]

Thanks for the info though!

[reubenmiller]

@dmydry @mingfang CodeceptJS v1.1.5 landed this morning so can you try it out again?