Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

isolated runner changes #6799

Merged
merged 126 commits into from Jun 16, 2020
Merged

isolated runner changes #6799

merged 126 commits into from Jun 16, 2020

Conversation

kuceb
Copy link
Contributor

@kuceb kuceb commented Mar 20, 2020

  • Closes N/A (internal only)

  • chunk of feat: support test retries #3968

  • cleanup cy/Cypress globals to allow testing cypress-in-cypress

  • add runner/integration - runner.spec.js that tests cypress runner as the AUT

  • add server/unit - reporter.spec.js that tests reporter from event snapshot data generated in runner/integration - runner.spec.js

Potential Tests to move into runner/integration

User facing changelog

N/A

Additional details

isolated runner:
yes:

  • covers everything but events to/from server
  • can snapshot mocha event data emitted to server-reporter
    • can be replayed to unit tests (e.g. server stdout reporter)
    • faster debugging / gives assurances to contract between server/browser
  • input real user spec code
    • (remove abstractions and test generation code - spec as string or file path)
    • (possibly reuse e2e test fixtures)
  • advanced browser UI features tied to spec code and user actions
    • (e.g print error to console, rerun test button, disable autoscrolling, selector playground, pin DOM node, hook failures w/ skipped tests, uncaught error outside test, clicking 'tests' button should emit event to electron)
    • example - testing hook failures
      • e2e test shares fixture spec w/ isolated runner test
        • e2e would cover stdout and exit code
        • isolated runner would cover UI / runner logic
        • optionally snapshot mocha event data from isolated runner
  • hard to test scenarios from driver/integration tests (testing reporter, testing result of test)
  • can test command log w/out affecting state (assertions on command log won't cause new command log)
    no:
  • not used to capture stdout (cannot test complex e2e scenario - multiple spec / compilation error / )
  • does not replace component tests
    misc:
  • think about extracting diff/snapshot utilities from isolated runner work
  • quick spike improve feedback loop on e2e testing (watch spec file)
  • think about adding visual regression testing to cypress for use in isolated runner tests
    • possibly move into cypress core
      implementation:
  • save/reload state should be one test
  • allow writing fixture test as string/file path not helper shape
  • move isolated logic into utility that can be required
  • move isolated specs to packages/runner/cypress/integration

How has the user experience changed?

Internal Only

  • added snapshot/diff utility with helpful diff error message
    20-03-31_14:09::41
    which is the result of this diff
    image

  • update snapshots by toggling this ui button
    20-03-31_14:09::04

PR Tasks

  • Have tests been added/updated?

@cypress-bot
Copy link
Contributor

cypress-bot bot commented Mar 20, 2020

Thanks for the contribution! Below are some guidelines Cypress uses when doing PR reviews.

  • Please write [WIP] in the title of your Pull Request if your PR is not ready for review - someone will review your PR as soon as the [WIP] is removed.
  • Please familiarize yourself with the PR Review Checklist and feel free to make updates on your PR based on these guidelines.

PR Review Checklist

If any of the following requirements can't be met, leave a comment in the review selecting 'Request changes', otherwise 'Approve'.

User Experience

  • The feature/bugfix is self-documenting from within the product.
  • The change provides the end user with a way to fix their problem (no dead ends).

Functionality

  • The code works and performs its intended function with the correct logic.
  • Performance has been factored in (for example, the code cleans up after itself to not cause memory leaks).
  • The code guards against edge cases and invalid input and has tests to cover it.

Maintainability

  • The code is readable (too many nested 'if's are a bad sign).
  • Names used for variables, methods, etc, clearly describe their function.
  • The code is easy to understood and there are relevant comments explaining.
  • New algorithms are documented in the code with link(s) to external docs (flowcharts, w3c, chrome, firefox).
  • There are comments containing link(s) to the addressed issue (in tests and code).

Quality

  • The change does not reimplement code.
  • There's not a module from the ecosystem that should be used instead.
  • There is no redundant or duplicate code.
  • There are no irrelevant comments left in the code.
  • Tests are testing the code’s intended functionality in the best way possible.

Internal

  • The original issue has been tagged with a release in ZenHub.

@kuceb kuceb mentioned this pull request Mar 20, 2020
57 tasks
@cypress
Copy link

cypress bot commented Mar 20, 2020



Test summary

7638 0 119 0


Run details

Project cypress
Status Passed
Commit 1a0ac78
Started Jun 16, 2020 5:58 PM
Ended Jun 16, 2020 6:06 PM
Duration 07:42 💡
OS Linux Debian - 10.1
Browser Multiple

View run in Cypress Dashboard ➡️


This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

@kuceb kuceb changed the title isolated runner changes [WIP] isolated runner changes Mar 23, 2020
@kuceb
Copy link
Contributor Author

kuceb commented Mar 23, 2020

ugh, this has-binary yarn.lock issue.

edit: needed to clear ~/.cache/yarn

@kuceb kuceb changed the title [WIP] isolated runner changes isolated runner changes Mar 23, 2020
@kuceb kuceb requested a review from brian-mann March 23, 2020 21:33
packages/driver/src/cypress/cy.js Outdated Show resolved Hide resolved
packages/server/test/unit/reporter.spec.js Outdated Show resolved Hide resolved
packages/server/test/unit/reporter.spec.js Outdated Show resolved Hide resolved
packages/driver/test/cypress/support/helpers.js Outdated Show resolved Hide resolved
Copy link
Member

@jennifer-shehane jennifer-shehane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason not to merge this into develop branch?

TODO: Running tests in runner needs instructions in the README.md, right now it only describes how to run unit tests - does not describe how to start server and run cypress tests.

I'm a little unclear about the overlap of these tests versus the reporter/cypress/integration tests. Are these meant to replace those entirely? Cause some of the things they're tested seem duplicated. Maybe I'm wrong here though.

packages/runner/test/cypress.json Outdated Show resolved Hide resolved
Comment on lines +221 to +227
// disable DOM snapshots during log for this agent
agent.snapshot = (bool = true) => {
agent._noSnapshot = !bool

return agent
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might be a better / more readable API to do...

agent.log = (options) => {
  if (options && options.snapshot === false) {
     // do the thing
  }
})

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or more like agent.options or agent.logOptions

Comment on lines 4 to 6
"hosts": {
"*.foobar.com": "127.0.0.1"
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove, copypasta fail

@@ -32,6 +32,12 @@ const socketRerunEvents = 'runner:restart watched:file:changed'.split(' ')
const localBus = new EventEmitter()
const reporterBus = new EventEmitter()

if (window.Cypress) {
window.channel = ws
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is already exposed via line 19

Also we may just want to nest these underneath window.runnerWs (and rename that)

window.eventManager = { backendBus, reporterBus, localBus }

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's add a note reflecting on why this is a weird shape - that we only want to expose this within the test environment, but because we use an e2e test that uses it, we either need to move that test to isolated-runner or find a 2nd way to expand the conditional.

}

describe('src/cypress/runner', () => {
describe('tests finish with correct state', () => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

runner.mochaEvents.spec move all the tests that snapshot mocha event structures to their own file

@@ -0,0 +1,54 @@
/**
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move this to a separate draft PR

.html(btnIcon.html().replace(/\n/g, '<br/>'))
},
click () {
const prev = Cypress.env('SNAPSHOT_UPDATE')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use Cypress.env, just put this on a global since we intend to move away from the current Cypress.env implementation

Comment on lines 70 to 71
}, function () {
})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lol 5th argument

@@ -1,60 +1,369 @@
exports['lib/reporter #stats has reporterName stats, reporterStats, etc 1'] = {
exports['simple_single_test runner emit'] = [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to draft PR

@@ -0,0 +1,124 @@
const _ = require('lodash')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to draft PBR

@@ -0,0 +1,92 @@
/* eslint-disable prefer-rest-params */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to draft PR

@@ -1,162 +1,248 @@
require('../spec_helper')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to draft PR

* @param {()=>void | {[key:string]: any}} mochaTests
* @param {{state?: any, config?: any}} opts
*/
const runCypress = (mochaTests, opts = {}) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

runIsolatedCypress

@kuceb kuceb requested a review from brian-mann June 16, 2020 15:08
@brian-mann brian-mann merged commit 6c6b23f into develop Jun 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants