Preprocessor swallows errors and ignores ENOENT for bundled files

[bahmutov]

bug

Cypress 3.0.1 on Mac

I have been playing with a spec preprocessor based on Parcel and noticed a curious thing. If the bundler is configured incorrectly and actually does NOT bundle files but returns a non-existing path, the test runner swallows the error and happily exits with code 0, while showing (and ignoring) ENOENT error

Example output

  Running: spec.js...                                                                      (1 of 1) 
bundle file once without watching to /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js
basename spec
bundle file once without watching to /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js
basename index
Error: ENOENT: no such file or directory, stat '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js'


Error: ENOENT: no such file or directory, stat '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js'


Error: ENOENT: no such file or directory, stat '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js'




  0 passing (2ms)


  (Results)

  ┌─────────────────────────┐
  │ Tests:        0         │
  │ Passing:      0         │
  │ Failing:      0         │
  │ Pending:      0         │
  │ Skipped:      0         │
  │ Screenshots:  0         │
  │ Video:        true      │
  │ Duration:     0 seconds │
  │ Spec Ran:     spec.js   │
  └─────────────────────────┘


  (Video)

  - Started processing:   Compressing to 32 CRF
  - Finished processing:  /Users/gleb/git/cypress-example-recipes/examples/blogs__parcel_preprocessor/cypress/videos/spec.js.mp4 (0 seconds)


====================================================================================================

  (Run Finished)


      Spec                                                Tests  Passing  Failing  Pending  Skipped 
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔ spec.js                                     3ms        -        -        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    All specs passed!                             3ms        -        -        -        -        -  

The bundles files are returned like this

const onFile = (file) => {
  const { filePath, shouldWatch, outputPath } = file
  if (!shouldWatch) {
    console.log('bundle file once without watching to %s', outputPath)
    return bundleOnce(filePath, outputPath).then(() => outputPath)
  }
  ...
}

Here is the relevant verbose logs

  cypress:server:socket watch test file integration/spec.js +0ms
  cypress:server:socket will watch test file path cypress/integration/spec.js +0ms
  cypress:server:preprocessor getFile /Users/gleb/git/cypress-example-recipes/examples/blogs__parcel_preprocessor/cypress/integration/spec.js +0ms
  cypress:server:appdata path: /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js +1ms
  cypress:server:plugins execute plugin event 'file:preprocessor' with args: EventEmitter { domain: null, _events: { rerun: [Function] }, _eventsCount: 1, _maxListeners: undefined, filePath: '/Users/gleb/git/cypress-example-recipes/examples/blogs__parcel_preprocessor/cypress/integration/spec.js', outputPath: '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js', shouldWatch: false } undefined undefined +0ms
  cypress:server:plugins call event file:preprocessor for invocation id inv1 +3ms
  cypress:server:plugins:child execute plugin event: file:preprocessor ({ eventId: 2, invocationId: 'inv1' }) +2s
bundle file once without watching to /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js
  cypress:server:controllers:spec request for cypress/support/index.js +12ms
  cypress:server:preprocessor getFile /Users/gleb/git/cypress-example-recipes/examples/blogs__parcel_preprocessor/cypress/support/index.js +1ms
  cypress:server:appdata path: /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js +0ms
  cypress:server:plugins execute plugin event 'file:preprocessor' with args: EventEmitter { domain: null, _events: { rerun: [Function] }, _eventsCount: 1, _maxListeners: undefined, filePath: '/Users/gleb/git/cypress-example-recipes/examples/blogs__parcel_preprocessor/cypress/support/index.js', outputPath: '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js', shouldWatch: false } undefined undefined +0ms
  cypress:server:plugins call event file:preprocessor for invocation id inv2 +1ms
  cypress:server:controllers:spec request for cypress/integration/spec.js +0ms
  cypress:server:preprocessor getFile /Users/gleb/git/cypress-example-recipes/examples/blogs__parcel_preprocessor/cypress/integration/spec.js +0ms
  cypress:server:preprocessor headless and already processed +0ms
  cypress:server:plugins:child execute plugin event: file:preprocessor ({ eventId: 2, invocationId: 'inv2' }) +52ms
bundle file once without watching to /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js
  cypress:server:plugins promise resolved for id 'inv1' with value /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js +444ms
  cypress:server:controllers:spec send /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js +0ms
Error: ENOENT: no such file or directory, stat '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js'


  cypress:server:plugins promise resolved for id 'inv2' with value /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js +5ms
  cypress:server:controllers:spec send /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js +0ms
Error: ENOENT: no such file or directory, stat '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js'


  cypress:server:controllers:spec request for cypress/integration/spec.js +2ms
  cypress:server:preprocessor getFile /Users/gleb/git/cypress-example-recipes/examples/blogs__parcel_preprocessor/cypress/integration/spec.js +0ms
  cypress:server:preprocessor headless and already processed +0ms
  cypress:server:controllers:spec send /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js +0ms
Error: ENOENT: no such file or directory, stat '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js'


  cypress:server:timers clearing timer id 11 from queue { '11': { args: [], ms: 85000, cb: [Function] } } +5ms
  cypress:server:timers queuing timer id 12 after 85000 ms +0ms
  cypress:server:timers child received timer id 12 +474ms
  cypress:server:timers clearing timer id 12 from queue { '12': { args: [], ms: 85000, cb: [Function] } } +9ms
  cypress:server:timers queuing timer id 13 after 85000 ms +0ms
  cypress:server:project onSetRunnables +1ms
  cypress:server:timers child received timer id 13 +10ms
  cypress:server:project runnables null +0ms
  cypress:server:reporter trying to load reporter: spec +0ms
  cypress:server:reporter spec is Mocha reporter +0ms
  cypress:server:timers clearing timer id 13 from queue { '13': { args: [], ms: 85000, cb: [Function] } } +12ms
  cypress:server:timers queuing timer id 14 after 85000 ms +0ms
  cypress:server:timers child received timer id 14 +12ms
  cypress:server:project onMocha start +0ms
  cypress:server:reporter got mocha event 'start' with args: [ { start: '2018-06-05T03:35:06.763Z' } ] +1ms

  cypress:server:timers clearing timer id 14 from queue { '14': { args: [], ms: 85000, cb: [Function] } } +2ms
  cypress:server:timers queuing timer id 15 after 85000 ms +0ms
  cypress:server:project onMocha end +0ms
  cypress:server:timers child received timer id 15 +3ms
  cypress:server:reporter got mocha event 'end' with args: [ { end: '2018-06-05T03:35:06.766Z' } ] +0ms

  0 passing (2ms)

the folder was there, just the file that was supposed to be there does not exist when we return it from preprocessor

[bahmutov]

Same problem happens on CI as well: https://circleci.com/gh/cypress-io/cypress-example-recipes/6174

[chrisbreiding]

Yeah, I guess we don't validate the path returned to make sure there's a file there. So it ends up 404ing in the browser which results in 0 tests. I suppose we should fs.stat the output path returned and fail if it's not there. But to some degree we need to trust that the preprocessor does the right thing (like erroring when it can't bundle). Because even if the file is there, it could be empty or not javascript or just not the right thing that should have been produced by your source spec file.

[bahmutov]

Yes, but we are ignoring the error which is the wrong way to go about it. External code should be least trusted in this case.

…

Sent from my iPhone

On Jun 5, 2018, at 13:45, Chris Breiding ***@***.***> wrote: Yeah, I guess we don't validate the path returned to make sure there's a file there. So it ends up 404ing in the browser which results in 0 tests. I suppose we should fs.stat the output path returned and fail if it's not there. But to some degree we need to trust that the preprocessor does the right thing (like erroring when it can't bundle). Because even if the file is there, it could be empty or not javascript or just not the right thing that should have been produced by your source spec file. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[chrisbreiding]

Figured out what's going on. The problem is in our controller code. res.sendFile takes a node-style callback but we're using it as if it returns a promise, so the error is getting swallowed. If I promisify that sendFile it properly errors and fails the test.

[bahmutov]

👍 great detective work. And the test did not catch this?

…

Sent from my iPhone

On Jun 5, 2018, at 14:00, Chris Breiding ***@***.***> wrote: Figured out what's going on. The problem is in our controller code. res.sendFile takes a node-style callback but we're using it as if it returns a promise, so the error is getting swallowed. If I promisify that sendFile it properly errors and fails the test. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[chrisbreiding]

We don't have a test concerning res.sendFile erroring.

[brian-mann]

Just need one e2e test covering this and I am happy

[brian-mann]

Released in 3.0.2.

[rovshenn]

I know this is a closed issue, but I am having this problem with custom commands using below versions:

"cypress": "^4.0.2",
"@nrwl/cypress": "8.5.2",

If I create a brand new project that has no custom commands, I see proper errors in the cli/headless run. If I have defined the support file: "supportFile": "./src/support/index.ts", I do not see any errors as I guess they get swollen as I see some path errors if I use node inpsect-brk, but the test show results success with nothing run.

Spec                                              Tests  Passing  Failing  Pending  Skipped
All specs passed!                          1ms        -        -        -        -        -

Also the process hangs saying:
Using 1 worker with 2048MB memory limit
and then after a min or so (during which time I assume it transpiles ts files) prints out above.

[rovshenn]

It's nrwl nx issue, if you upgrade to 8.12.10 (latest ver at this time), headless run works as expected. Leaving here if someone else is having this issue