Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Capture + Display all logs for everything that happens in Cypress #448

Open
brian-mann opened this issue Mar 9, 2017 · 51 comments
Open

Capture + Display all logs for everything that happens in Cypress #448

brian-mann opened this issue Mar 9, 2017 · 51 comments

Comments

@brian-mann
Copy link
Member

@brian-mann brian-mann commented Mar 9, 2017

Problem

Many of our users express the desire to see console.log statements when running headlessly. Typically you'd be able to see these when you're working in Dev Tools (in headed GUI mode).

We can pretty easily display console.log statements by mixing them into stdout when running headlessly, but we don't think this is really a "true" solution. It's kind of a hack and it still doesn't really tell you "everything" that's happening. For instance, when a console.log happens, how would you know when it happened in relation to everything else that's going on in a test?

So, we think to truly solve this problem, we need a comprehensive way of displaying not just console.log statements but also capturing:

  • Network Traffic
  • Commands
  • Retries
  • Under the hood Cypress wizardry
  • DOM snapshots on failure

By providing all of this, the entire picture of what happened during a test is revealed and now you actually have even more information at your disposal than even what the GUI headed version provides you.

Solution

We're going to make these logs available both in our Dashboard, and in the Desktop App itself.

Once we capture and parse these logs, it will unlock an enormous amount of power in Cypress. We'll also be able to analyze the logs and deliver things like:

  • Analytics into your tests
  • Comparing a failing test to a previously passing one
  • Diffing the logs for changes in response bodies, etc
  • Understanding why tests are slow
  • Discovering command anti patterns (like explicit waits in your code)

The logs could also be interactive - for instance they should "sync" up to the video and enable you to playback everything that actually happened in a Cypress test.

Here is an example comp of what we're going for.

logs 1

We're still working on communicating the start and end of logs, and also communicating that a log event acts as "a stream" but that events are connected to other events.

For instance when a HTTP request starts - that's a 'start' event. When the response comes back, thats a separate 'end' event. However these are connected and you likely need to be able to view the data for both events from either the start or end event.

Another example:

A command starts. It retries 65x times. It eventually finishes. The duration of the retry events, the reason its retrying, and the eventual delta between a start and an end needs to be communicated intelligently.

These log files end up being a massive amount of data, but we already have the infrastructure in place to capture, parse and deliver them. At this point we're just iterating on a UI that's intuitive and doesn't overwhelm the user.

We're going to be providing a new Logs tab, but also for individual test failures, we'll just be providing the logs for that one independent test.

Feedback

...is welcome ;-)

@brian-mann
Copy link
Member Author

@brian-mann brian-mann commented Mar 9, 2017

@bahmutov what do you think?

@bahmutov
Copy link
Contributor

@bahmutov bahmutov commented Mar 9, 2017

Perfecto! 😍

@ghost
Copy link

@ghost ghost commented Mar 16, 2017

@brian-mann excellent. Really looking forward to getting this. We're spending lots of time trying to debug why our compiled JS is not working when served static via Cypress. We cannot replicate anywhere else, so one peek at the logs would like solve everything!

@danielnass
Copy link

@danielnass danielnass commented Jul 14, 2017

Hi guys!

Amazing feature! Any news about that?

Thanks

@cristopher-rodrigues
Copy link

@cristopher-rodrigues cristopher-rodrigues commented Jul 14, 2017

AMAZING!

@lukemadera
Copy link

@lukemadera lukemadera commented Oct 2, 2017

@brian-mann I've been banging my head against a wall trying to upgrade to cypress 0.20.1. It's been days - things work fine in the GUI but fail from the console. And not being able to log anything is infuriating. After super old school commenting out code one line at a time until it passes again to isolate the line, I've found that it is cy.request that fails. But I don't know how to fix it yet, and I can't log anything so have no idea what the issue is. Adding all the features you noted in this ticket sounds fine, but we need a simple way to debug locally, in the command line itself. How can we get console.log (or cy.log) to show in stdout when running headlessly? We're blocked from using Cypress right now until we fix that because our CI is failing now due to this other error, which is supposedly fixed in 0.20 #517

@bahmutov

This comment was marked as off-topic.

@brian-mann

This comment was marked as off-topic.

@brian-mann

This comment was marked as off-topic.

@brian-mann

This comment was marked as off-topic.

@lukemadera

This comment was marked as off-topic.

@brian-mann

This comment was marked as off-topic.

@brian-mann
Copy link
Member Author

@brian-mann brian-mann commented Oct 2, 2017

Also at this very moment you could do this trick here: #300 (comment)

That will get all console.log's to show up in the command log, which will stringify and print their arguments. You could then watch the video / screenshot to see what the information is.

That's what I would do today when running in CI. Of course locally just show the browser and iterate on it there.

@brian-mann

This comment was marked as off-topic.

@lukemadera

This comment was marked as off-topic.

@brian-mann

This comment was marked as off-topic.

@rovansteen
Copy link

@rovansteen rovansteen commented Oct 16, 2017

This would be a really nice addition. Right now my tests are failing on the CI and it looks like it has something to do with cookies but because Cypress is basically a black box right now on CI I have no way of figuring out what's going on. Hope this feature will arrive soon, keep up the good work 👏🏼

@brian-mann
Copy link
Member Author

@brian-mann brian-mann commented Oct 17, 2017

Related to #699 and #700.

Those will land pretty soon and should help alleviate these problems.

@brandonb927
Copy link

@brandonb927 brandonb927 commented Dec 15, 2017

Just want to throw my 2 cents in, I'm also having an issue with (what I can only imagine is cookie-related) the CI build vs local cypress open run that @rovansteen is having, and not having a way to see what network requests are made is making it challenging to debug.

So far I'm loving Cypress though, you folks are doing amazing work and I'm more than willing to fork over our company credit card for a paid plan in the future!

@Pomax
Copy link

@Pomax Pomax commented Jan 31, 2019

The dashboard is not available during automated testing runs, such as running Cypress on Travis-CI, where it is critically important that there is a way to be able to log output to stdout so that it's possible to track where in a script things start to go wrong.

Can we still get something like cy.stdout.write?

@jennifer-shehane
Copy link
Member

@jennifer-shehane jennifer-shehane commented Feb 1, 2019

Hey @Pomax

The dashboard is not available during automated testing runs, such as running Cypress on Travis-CI

Could you explain a bit more by what you mean in the statement above? Is your company not able to record to the Dashboard? I know there have been requests for On Prem due to security restrictions, for example.

Having the ability to access the raw logs is certainly something our team is considering.

@Pomax
Copy link

@Pomax Pomax commented Feb 1, 2019

We're using Cypress during our Travis-CI passes, so Cypress gets run in a fully contained test environment with only "stdout output" as viewable data. Cypress gets to run the interaction tests, and take snapshots, but without access to any kind of external app. because it needs to be able to do its work irrespective of whether "the rest of the internet" is up or not.

I don't really want "access to logs", if those are log files written to some particular location, because the nature of CI is that you get to run code, and only run code. Everything is thrown away the moment testing is done, and the only thing that gets preserved is the console output.

So there really should be a way to say something like , for instance, cy.console.log("...") or cy.stdout.write("...") or something, so that people can get data written to the running console output directly.

Compare this output to the next:

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    3.0.3                                                                              │
  │ Browser:    Electron 59 (headless)                                                             │
  │ Specs:      2 found (multi-page.js, single-page.js)                                            │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────

  Running: multi-page.js...                                                                (1 of 2)


  multipage visual regression tests
2019-02-01 09:51:41,051 [INFO] "GET /en/campaigns/multi-page/ HTTP/1.1" 200 13332
2019-02-01 09:51:41,051 [INFO] "GET /en/campaigns/multi-page/ HTTP/1.1" 200 13332
2019-02-01 09:51:41,182 [INFO] "GET /_images/green-20.svg HTTP/1.1" 200 3459
[...snip...]
2019-02-01 09:51:41,510 [INFO] "GET /environment.json HTTP/1.1" 200 235
2019-02-01 09:51:41,510 [INFO] "GET /environment.json HTTP/1.1" 200 235
    √ Loads the multipage campaign correctly (4415ms)


  1 passing (4s)


  (Results)

  ┌─────────────────────────────┐
  │ Tests:        1             │
  │ Passing:      1             │
  │ Failing:      0             │
  │ Pending:      0             │
  │ Skipped:      0             │
  │ Screenshots:  0             │
  │ Video:        false         │
  │ Duration:     4 seconds     │
  │ Spec Ran:     multi-page.js │
  └─────────────────────────────┘

The above run tells use nothing about where in the test things are happening. Being able to effect something like the following (without resorting to custom plugin interactions) would be far more valuable to have when doing automated testing (which is of course the whole reason people start using Cypress: they have testing to automate).

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    3.0.3                                                                              │
  │ Browser:    Electron 59 (headless)                                                             │
  │ Specs:      2 found (multi-page.js, single-page.js)                                            │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────

  Running: multi-page.js...                                                                (1 of 2)


  multipage visual regression tests
2019-02-01 09:51:41,051 [INFO] "GET /en/campaigns/multi-page/ HTTP/1.1" 200 13332
2019-02-01 09:51:41,051 [INFO] "GET /en/campaigns/multi-page/ HTTP/1.1" 200 13332
[CYPRESS:multi-page.js:21] Page loaded, waiting for React bundle...
[CYPRESS:multi-page.js:24] React bundle loaded
[CYPRESS:multi-page.js:28] Running Signup Form interaction tests
2019-02-01 09:51:41,182 [INFO] "GET /_images/green-20.svg HTTP/1.1" 200 3459
[...snip...]
[CYPRESS:multi-page.js:112] Testing 2FA authentication dialog
[CYPRESS:multi-page.js:118] ERROR: undefined is not true
[...snip...]
2019-02-01 09:51:41,510 [INFO] "GET /environment.json HTTP/1.1" 200 235
2019-02-01 09:51:41,510 [INFO] "GET /environment.json HTTP/1.1" 200 235
    √ Loads the multipage campaign correctly (4415ms)


  6 passing (4s), 1 failed (4s)


  (Results)

  ┌─────────────────────────────┐
  │ Tests:        7             │
  │ Passing:      6             │
  │ Failing:      1             │
  │ Pending:      0             │
  │ Skipped:      0             │
  │ Screenshots:  0             │
  │ Video:        false         │
  │ Duration:     4 seconds     │
  │ Spec Ran:     multi-page.js │
  └─────────────────────────────┘

Using the dashboard for "in house" or local dev work is perfectly fine, but paid automated CI testing means everyone is running cypress without a dashboard. Not out of spite, and not because "they hate having to use a GUI" but because they literally can't run any graphical applications because the testing environment itself is fully automated, and everything in it gets thrown away (which a good CI service should be doing) leaving you just with the CI run console output to see what went wrong, if something went wrong.

@mitar
Copy link

@mitar mitar commented Mar 2, 2019

OK, while we wait for this perfect solution described in OP to arrive, I would really like to vote for a very simple solution of redirecting all browser console to the terminal during CI run. It is really hard to debug anything currently when it fails in CI. I would love to simple see in terminal all logging calls I might have inside tests themselves and also from my app. I mean, we already have those logging calls for debugging with a browser, let's just log them to the terminal as well.

Let's not make the perfect the enemy of the good.

@jennifer-shehane
Copy link
Member

@jennifer-shehane jennifer-shehane commented Mar 8, 2019

@bahmutov created a plugin that will print Cypress commands to the terminal - not exactly the browser console logs wanted, but maybe will help someone. https://github.com/bahmutov/cypress-failed-log

We've also been expanding on our Debugging doc - adding some weird tricks in there that could help some people. https://on.cypress.io/debugging

@Knaledge
Copy link

@Knaledge Knaledge commented Apr 1, 2019

Adding to the general commentary above: we've also run into this, namely the lack of full Cy-related transactions not being made available to some target (like stdout) when running headlessly/headlessly in CI (Jenkins).

In our case. where this really becomes a pain is when things pass "locally" - but fail in CI, or fail in Docker Compose scene (and running in a container with open instead of run is already classed as "not supported").

The sooner this gets introduced - even in some MVP form - the better.

@aboutlo
Copy link

@aboutlo aboutlo commented Apr 4, 2019

First apologies for the comment which is not trying to fix the actual problem. 🙏
Then let's say cypress is really cool and I really liked to use it 🙌

However, I can't suggest / use it to anyone until this is fix it.
The workaround is fine but far from being a good solution.
Being by default blind in CI is a big deal.

@MaaikeFox
Copy link

@MaaikeFox MaaikeFox commented May 29, 2019

What I now do is keep cy.logs really short so that I can hopefully read them from the screenshots that are available on the Dashboard. But it is a silly solution of course. It would be so helpful if the cy.logs would be available on the Dashboard

@saas2813
Copy link

@saas2813 saas2813 commented Jun 17, 2019

I want to assert that the Console log is empty, or only contains previously approved lines in the CI build. The optimal implementation would give a simple textfile with timestamped direct copies from the console output.
These logs could be prepended by the testcase name and current URL for context, adding more stuff would be noice.

As others have commented, it feels rather bad that the console log and Cypress command log can not be saved to files (separately or together, by configuration) 2.5 years after the issue (#300) came up.

@rodya-mirov
Copy link

@rodya-mirov rodya-mirov commented Jun 27, 2019

It's been 2.5 years since this was brought up and the fix everyone wants was declared "easy." All of the suggested fixes around using the dashboard are problematic for reasons described thoroughly in this issue. Is there any movement on this?

@bahmutov
Copy link
Contributor

@bahmutov bahmutov commented Jun 27, 2019

@alfaproject
Copy link

@alfaproject alfaproject commented Jul 14, 2019

Just stumbled upon this need ):

@waitsangbv
Copy link

@waitsangbv waitsangbv commented Aug 1, 2019

https://github.com/flotwig/cypress-log-to-output workaround only works on Chrome. But chrome headless does not work and does not support video recording... its going round and round the loop for a very simple request (i.e. proper console,log)

@hvuong-sigma
Copy link

@hvuong-sigma hvuong-sigma commented Aug 13, 2019

@jennifer-shehane

I don't see the feature listed in the Cypress Roadmap anymore and the two suggested workarounds aren't ideal, at least for me. Is there still planned work for this? We're fully integrated with Cypress and Cypress Dashboard but we're having trouble debugging issues without having the console.log available when running tests via CI.

@RamiaSaidawi
Copy link

@RamiaSaidawi RamiaSaidawi commented Oct 11, 2019

any update on how we can get console.log showing when we use "cypress run" ?
e are thinking of moving to cypress and this can be a show stopper :(

@x-yuri
Copy link

@x-yuri x-yuri commented Dec 7, 2019

In case you're using Electron browser you can see your console.log() statements with DEBUG=cypress:electron npx cypress run. More on it here. For Chrome the abovementioned cypress-log-to-output plugin should help

@craig-dae
Copy link

@craig-dae craig-dae commented Mar 6, 2020

Any updates on this? I have a cicd test that fails, that I can't reproduce locally, and I'm burning through my Github Action minutes.

@jennifer-shehane
Copy link
Member

@jennifer-shehane jennifer-shehane commented Apr 14, 2020

This issue is still in the 'proposal' stage, which means no work has been done on this issue as of today, so we do not have an estimate on when this will be delivered.

Current workarounds: use these plugins

@AbrahamBrookes
Copy link

@AbrahamBrookes AbrahamBrookes commented Mar 12, 2021

I'd like to write my own little helper for this, what command do I use to print arbitrary strings to the CLI when using cypress run? Or is that the underlying issue here - that there is none?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet