Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: only take snapshots if the AUT document is in state #24519

Merged
merged 12 commits into from Nov 7, 2022

Conversation

AtofStryker
Copy link
Contributor

@AtofStryker AtofStryker commented Nov 3, 2022

User facing changelog

Interactive time travel snapshots no longer capture all of Cypress application when Application Under Test frame has navigated away.

Additional details

Main thing this is fixing is new behavior introduced in 10.9.0 likely with #23297 where there is now async behavior from when snapshots are taken vs what document is loaded in state. This was leading to Cypress in Cypress like snapshots, which don't provide the user a lot of value and are confusing. This PR makes sure either the document is in state or the snapshot is coming from a cross origin spec bridge (cy.origin).

Steps to test

How has the user experience changed?

before fix

snapshot-before-fix

after fix

snapshot-after-fix

PR Tasks

  • Have tests been added/updated? (tests to make sure no regressions in snapshot behavior are passing)
  • Has the original issue (or this PR, if no issue exists) been tagged with a release in ZenHub? (user-facing changes only)
  • Has a PR for user-facing changes been opened in cypress-documentation?N/A
  • Have API changes been updated in the type definitions?

@cypress-bot
Copy link

cypress-bot bot commented Nov 3, 2022

Thanks for taking the time to open a PR!

@AtofStryker AtofStryker self-assigned this Nov 3, 2022
@AtofStryker AtofStryker marked this pull request as ready for review November 3, 2022 14:50
@cypress
Copy link

cypress bot commented Nov 3, 2022



Test summary

422 0 10 0Flakiness 2


Run details

Project cypress
Status Passed
Commit 6f61da3
Started Nov 7, 2022 6:30 PM
Ended Nov 7, 2022 6:41 PM
Duration 11:37 💡
OS Linux Debian -
Browser Chrome 106

View run in Cypress Dashboard ➡️


Flakiness

runs.cy.ts Flakiness
1 App: Runs > Runs - Login > when logged out, shows call to action
specs_list_latest_runs.cy.ts Flakiness
1 App/Cloud Integration - Latest runs and Average duration > when no runs are recorded > shows placeholders for all visible specs

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

Copy link
Contributor

@chrisbreiding chrisbreiding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to add a test for this?

@AtofStryker
Copy link
Contributor Author

Is it possible to add a test for this?

I think we can but I am not sure it will be reliable. Maybe a test that executes a visit away from the primary domain and checks the log snapshot to see if one exists, and if one does, fail the test?

@chrisbreiding
Copy link
Contributor

I think we can but I am not sure it will be reliable. Maybe a test that executes a visit away from the primary domain and checks the log snapshot to see if one exists, and if one does, fail the test?

It doesn't necessarily have to be a full e2e test. More of unit/integration test like these would still give us some coverage.

@AtofStryker
Copy link
Contributor Author

I think we can but I am not sure it will be reliable. Maybe a test that executes a visit away from the primary domain and checks the log snapshot to see if one exists, and if one does, fail the test?

It doesn't necessarily have to be a full e2e test. More of unit/integration test like these would still give us some coverage.

sounds good to me. I added a small integration test to make sure the snapshots are null after the document unloads as well as an e2e test to make sure cy origin snapshots still work in 97da4aa

packages/driver/src/cy/snapshots.ts Outdated Show resolved Hide resolved
AtofStryker and others added 2 commits November 4, 2022 09:54
Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com>
if (!preprocessedSnapshot && !state('document')) {
return null
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this fix just mean there are more missing screenshots? Can you share a few scenarios this is currently broken? I'd like to test the before & after locally. Does the comment on line 249 no long valid? This is where we are falling back to window.document....would it be more appropriate to query for the AUT iframe?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am also wondering if we should update some of our origin tests to verify there are snapshots / no snapshots for various scenarios.

I didn't realize the updated tests were specific to origin tests for snapshot. Will be exicted to merge these someday 😄

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the comment on line 249 no long valid? This is where we are falling back to window.document

That's a good question. Seems like these lines should be updated/removed.

would it be more appropriate to query for the AUT iframe?

I think if there's no state('document'), it won't be possible because that means we don't have synchronous access to the AUT.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This actually brings up a good point, but I think it needs to stay. Main reason is logs streaming in from the secondary need to be cloned off a document, which in this case the primary document state is unloaded. But the backup isn't the AUT document in this case, rather the actual top document, which is OK since we only using the window to clone the node. If anything, we should probably use window.document here instead of state("document")

I'll play around with this to make sure it works as expect and will add a comment to explain this logic

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this fix just mean there are more missing screenshots? Can you share a few scenarios this is currently broken? I'd like to test the before & after locally.

@emilyrohrbough you can run the snapshot tests from the driver locally, but the impact isn't exactly super noticeable there. Do you have cypress-origin-providers set up at all?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added comments in 228cef0. I kept the current logic of using loaded document in state first before the specbridge/root document because of how many edge cases exist in snapshot code that could potentially break, but not be caught by this change. Here be 🐉 s.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can easily setup cypress-origin-providers but even if I run the driver tests, what would I be looking for to verify this was acutally fixed? xhr requests?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you run the snapshots.cy.ts inside the origin driver tests, hover over the (page load) or (new url) logs to see no snapshot taken, where as before it would look something like this:
Screen Shot 2022-11-04 at 1 07 54 PM


expect(xhrLogFromSecondaryOrigin).to.not.be.undefined
expect(getLogFromSecondaryOrigin).to.not.be.undefined
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
expect(getLogFromSecondaryOrigin).to.not.be.undefined
expect(getLogFromSecondaryOrigin).to.be.defined

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure why but I cannot get this type to show up in a chai assertion. I don't think it is actually valid 🙁

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, doesn't seem to be a bundled chai assertion, but we add on to chai with various libs, so it's possible it ends up as an assertion somehow, since we do use to.be.defined in a few other tests. I think the "correct" assertion is .to.exist.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated to to.exist in 17d37f5


beforeEach(() => {
logs = new Map()
return (props?.consoleProps?.Selector === selector)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What log is this associated with?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case ,we are just digging out the [data-cy="assertion-header"] get log to make sure the snapshot is sent back over and not omitted with our new logic changes

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that associated with the origin command log or an xhr log?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

origin command log since the get is invoked inside the cy.origin

})

cy.visit('http://www.foobar.com:3500/fixtures/xhr-fetch-requests.html')
it('Does not take snapshots of XHR/fetch requests from secondary origin if the wrong origin is / origin mismatch, but instead the primary origin (existing behavior)', {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read this 2x and am a little confused

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is missing the word visited in the name. I can add it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in 1d43234

Cypress.config('isInteractive', true)

cy.get(`[data-cy="assertion-header"]`).should('exist')
cy.wait('@fooBarBaz')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this failing because the alias does not exist?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NVM coming from beforeEach

})

// TODO: fix flaky test https://github.com/cypress-io/cypress/issues/23437
it.skip('verifies fetch requests made while a secondary origin is active eventually update with snapshots of the secondary origin', () => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we able to unskip these tests? Maybe add some retries?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me revisit this and see what the current state is

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I can get the tests to pass, but from the flake report the flake is fairly rare, like 2%. I think by default we already retry twice in CI, so I doubt it will fix the problem immediately. @mjhenkes and I were able to reproduce this a few months back, and if I remember correctly there were collisions with the proxy logging on xhr request. I could be completely wrong though. It's been a while 😅

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's possible this gets resolved by removing server/route, but i'm not 100%. I vote we revisit the flake tests after 12.0.0

@AtofStryker
Copy link
Contributor Author

@chrisbreiding @emilyrohrbough this should be ready for another look. I forgot I needed to set isInteractive inside cy.origin to receive snapshots

@AtofStryker AtofStryker merged commit 3cdbeca into develop Nov 7, 2022
80 of 83 checks passed
@AtofStryker AtofStryker deleted the bugfix/cross-origin-snapshots branch November 7, 2022 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve cy.origin time travel snapshots
4 participants