New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Capture + Display all logs for everything that happens in Cypress #448

Open
brian-mann opened this Issue Mar 9, 2017 · 29 comments

Comments

@brian-mann
Member

brian-mann commented Mar 9, 2017

Problem

Many of our users express the desire to see console.log statements when running headlessly. Typically you'd be able to see these when you're working in Dev Tools (in headed GUI mode).

We can pretty easily display console.log statements by mixing them into stdout when running headlessly, but we don't think this is really a "true" solution. It's kind of a hack and it still doesn't really tell you "everything" that's happening. For instance, when a console.log happens, how would you know when it happened in relation to everything else that's going on in a test?

So, we think to truly solve this problem, we need a comprehensive way of displaying not just console.log statements but also capturing:

  • Network Traffic
  • Commands
  • Retries
  • Under the hood Cypress wizardry
  • DOM snapshots on failure

By providing all of this, the entire picture of what happened during a test is revealed and now you actually have even more information at your disposal than even what the GUI headed version provides you.

Solution

We're going to make these logs available both in our Dashboard, and in the Desktop App itself.

Once we capture and parse these logs, it will unlock an enormous amount of power in Cypress. We'll also be able to analyze the logs and deliver things like:

  • Analytics into your tests
  • Comparing a failing test to a previously passing one
  • Diffing the logs for changes in response bodies, etc
  • Understanding why tests are slow
  • Discovering command anti patterns (like explicit waits in your code)

The logs could also be interactive - for instance they should "sync" up to the video and enable you to playback everything that actually happened in a Cypress test.

Here is an example comp of what we're going for.

logs 1

We're still working on communicating the start and end of logs, and also communicating that a log event acts as "a stream" but that events are connected to other events.

For instance when a HTTP request starts - that's a 'start' event. When the response comes back, thats a separate 'end' event. However these are connected and you likely need to be able to view the data for both events from either the start or end event.

Another example:

A command starts. It retries 65x times. It eventually finishes. The duration of the retry events, the reason its retrying, and the eventual delta between a start and an end needs to be communicated intelligently.

These log files end up being a massive amount of data, but we already have the infrastructure in place to capture, parse and deliver them. At this point we're just iterating on a UI that's intuitive and doesn't overwhelm the user.

We're going to be providing a new Logs tab, but also for individual test failures, we'll just be providing the logs for that one independent test.

Feedback

...is welcome ;-)

@brian-mann

This comment has been minimized.

Member

brian-mann commented Mar 9, 2017

@bahmutov what do you think?

@bahmutov

This comment has been minimized.

Collaborator

bahmutov commented Mar 9, 2017

Perfecto! 😍

@ghost

This comment has been minimized.

ghost commented Mar 16, 2017

@brian-mann excellent. Really looking forward to getting this. We're spending lots of time trying to debug why our compiled JS is not working when served static via Cypress. We cannot replicate anywhere else, so one peek at the logs would like solve everything!

This was referenced Mar 21, 2017

@danielnass

This comment has been minimized.

danielnass commented Jul 14, 2017

Hi guys!

Amazing feature! Any news about that?

Thanks

@cristopher-rodrigues

This comment has been minimized.

cristopher-rodrigues commented Jul 14, 2017

AMAZING!

@jennifer-shehane jennifer-shehane referenced this issue Sep 6, 2017

Open

Add FAQ questions from gitter #25

10 of 32 tasks complete
@lukemadera

This comment has been minimized.

lukemadera commented Oct 2, 2017

@brian-mann I've been banging my head against a wall trying to upgrade to cypress 0.20.1. It's been days - things work fine in the GUI but fail from the console. And not being able to log anything is infuriating. After super old school commenting out code one line at a time until it passes again to isolate the line, I've found that it is cy.request that fails. But I don't know how to fix it yet, and I can't log anything so have no idea what the issue is. Adding all the features you noted in this ticket sounds fine, but we need a simple way to debug locally, in the command line itself. How can we get console.log (or cy.log) to show in stdout when running headlessly? We're blocked from using Cypress right now until we fix that because our CI is failing now due to this other error, which is supposedly fixed in 0.20 #517

@bahmutov

This comment has been minimized.

Collaborator

bahmutov commented Oct 2, 2017

@lukemadera have you tried using DEBUG variable to see messages from Cypress? https://docs.cypress.io/guides/guides/debugging.html#Debug-the-Command-Line

@brian-mann

This comment has been minimized.

Member

brian-mann commented Oct 2, 2017

  1. Is it failing locally or only in CI?
  2. If failing locally be sure to run in --headed mode so you can see what's going on.
  3. You can also switch to using --browser chrome from the CLI (in local and in CI) if it fails only in electron. Electron is technically a different browser from your normal Chrome.
  4. The key here is getting it to be reproduced locally. You can try running Chrome from the GUI, but using the Dev Tools to throttle back the network, and also decrease the CPU so that it mimics a much slower computer. That can sometimes highlight a race condition you are not accounting for.
  5. As per getting events out of the browser - that is not necessarily a straightforward thing. What exactly would you like us to send to stdout? Internal information about what Cypress is doing? You will see hundreds if not thousands of events being fired over the course of a run. And that's just what Cypress is doing. Nothing from your application like network requests, console output, etc would be sent unless we chose to do that.
  6. I would like to run an experiment here. Open up Cypress in GUI mode and then open the console and follow the instructions here: https://docs.cypress.io/api/events/catalog-of-events.html#Logging-All-Events.

That will send every internal event that Cypress fires to the console. Look at this and tell me if it is helpful. If the answer is yes, we could pretty easily get that out of the browser and into stdout. If it is not helpful then we'd need to figure out what is. Browsers are enormously complex and trying to distill what's happening down into text is not that simple. What exactly do you want to see? What would be helpful?

@brian-mann

This comment has been minimized.

Member

brian-mann commented Oct 2, 2017

Also one last important piece of information. Why aren't you just recording your runs? Even though its headless you can see exactly what is going on since we capture a movie and screenshots. What about that information is not enough to see what's going on and be able to reproduce locally?

@brian-mann

This comment has been minimized.

Member

brian-mann commented Oct 2, 2017

Or at the very least why aren't you watching the video from headless runs or looking at screenshots?

@lukemadera

This comment has been minimized.

lukemadera commented Oct 2, 2017

@bahmutov Yes I've tried debug. That gives general info, not info I generate and want to see.

@brian-mann thanks for the speedy response. There are 2 failures / issues right now.
A. on CI, v0.19.x no longer runs (headless)
B. locally, v0.20.x runs in GUI but fails in headless.

  1. All the tests pass in the GUI so unless I'm missing something, the error won't happen in headed mode, so I can only debug in headless mode.

  2. Again, it's a general request issue (I think - still need logging), not a browser issue.

  3. I can reproduce it locally 100% of the time, in headless mode. Works 100% of the time in GUI mode. These are tests that worked fine for months, and they work in v0.19. Just not when upgrading to v0.20.x

  4. I'm simply looking to be able to do console.log() and have it show up on the command line when I do ./node_modules/.bin/cypress run --spec cypress/integration/locations/locations-list.js

  5. while all that data MAY be helpful, right now I just need ONE (or a few) pieces of data, that I choose from console.log()

  6. I'm not even getting to the screenshots or videos really, and I've already figured out the problem (it's in our data setup phase, trying to login via cy.request). It's not an actual test that is failing, we're not even making it to the tests. So I'm not sure how videos / screenshots would help. It's literally the first step that's failing, in setup, before tests even start.

If this is still unclear I can record a video of the test in the GUI and the test in the command line - one finishes and passes, one just gets stuck and never does anything (or times out).

Update: here's the video: http://recordit.co/DdfW8hgK74

@brian-mann

This comment has been minimized.

Member

brian-mann commented Oct 2, 2017

Okay let me clear up a few things.

When we say GUI mode we're talking about launching cypress from cypress open. In that mode the tests never "complete" and you don't get anything on stdout.

When you run from the CLI via cypress run then the tests complete, you get stdout and an exit code.

Cypress itself does work differently in these two modes. When running from the CLI it makes internal optimizations to be able to run all the tests - ie it will run faster than the GUI mode. The GUI mode is focused on debuggability and interaction.

My point is this: using --headed from the CLI simply shows you the browser. It will behave identically to running headless. One shows the Electron browser window, and the other doesn't. That's it.

In fact you could just launch Chrome via the CLI with --browser chrome and then you'll be using the same two browsers in GUI mode and in CLI mode with the only difference that one runs to completion, the other doesn't.

The screenshots and video would still help in this situation, albeit I understand the need for console.log. At the very least they would show you precisely which step and command is failing without you having to guess in the dark.

Update: per your video - just add the --headed option or use --browser chrome to see what's going on. If you choose not to do that, you could still use the video Cypress captures when you run headlessly so you could still see what happened in that mode.

@brian-mann

This comment has been minimized.

Member

brian-mann commented Oct 2, 2017

Also at this very moment you could do this trick here: #300 (comment)

That will get all console.log's to show up in the command log, which will stringify and print their arguments. You could then watch the video / screenshot to see what the information is.

That's what I would do today when running in CI. Of course locally just show the browser and iterate on it there.

@brian-mann

This comment has been minimized.

Member

brian-mann commented Oct 2, 2017

I opened a couple new issues to do what you're asking with exporting the console output, and some improvements to the debugging the driver.

@lukemadera

This comment has been minimized.

lukemadera commented Oct 2, 2017

YES! The --headed works and I can see console.log output, thanks so much @brian-mann ! It does still close automatically at the end, which means I can't see the logs persisted, so I have to add a wait at the end. Any way to prevent the browser from auto-closing?

However, I'm still stuck at the same cy.request line. It just doesn't ever complete. In digging into it, cy.request works fine if it's called BEFORE cy.readFile. If it's used after (within the .then()), it just never returns.
In trying to reproduce it, if I simplify things it seems to work, so I'm not clear on what the issue is.
If I just do a request (using the npm request module instead of cy.request, I get a CORS error).`

Luckily, the request I'm attempting is to logout so in this case I can just skip that request all together, and then things work fine. Now tests appear to be passing again.. Locally at least. Going to try in CI now.

Not sure if the issue is "fixed" but it's working for me now it seems. Thanks for all the speedy responses and help!

@brian-mann

This comment has been minimized.

Member

brian-mann commented Oct 2, 2017

If you want to iterate without it close, just open up Electron in the GUI. Just select that browser instead of Chrome. You may see the same issue.

As I mentioned if that's the case just switch to booting chrome from the CLI --browser chrome

@rovansteen

This comment has been minimized.

rovansteen commented Oct 16, 2017

This would be a really nice addition. Right now my tests are failing on the CI and it looks like it has something to do with cookies but because Cypress is basically a black box right now on CI I have no way of figuring out what's going on. Hope this feature will arrive soon, keep up the good work 👏🏼

@brian-mann

This comment has been minimized.

Member

brian-mann commented Oct 17, 2017

Related to #699 and #700.

Those will land pretty soon and should help alleviate these problems.

@brandonb927

This comment has been minimized.

brandonb927 commented Dec 15, 2017

Just want to throw my 2 cents in, I'm also having an issue with (what I can only imagine is cookie-related) the CI build vs local cypress open run that @rovansteen is having, and not having a way to see what network requests are made is making it challenging to debug.

So far I'm loving Cypress though, you folks are doing amazing work and I'm more than willing to fork over our company credit card for a paid plan in the future!

@RRMoelker

This comment has been minimized.

RRMoelker commented Feb 14, 2018

Hi, loving Cypress. But I am curious. Are these issues still being addressed?

I see a lot of comments that these issues are being worked on, like: "x and y. Those will land pretty soon". But this seems to have been in development for nearly a year now.

Respectfully, if we can't debug our CI than this isn't really a viable product for the long term. And we would rather know sooner than later since E2E tests are very costly.

@aboutlo

This comment has been minimized.

aboutlo commented Apr 4, 2018

@brian-mann do you have any update on this one? In October you said soon. I know your backlog is very long but CI logs are quite important. Every time there is an issue on the CI we need to run the tests again locally in order to get the logs.
Thank you for your understanding 🙏

@szabyg

This comment has been minimized.

szabyg commented May 7, 2018

I'm wondering if anything is happening here, it's way too quite for a feature all projects need at the point where they have to maintain tests and the CI console output and the test results can't tell what went wrong except the last exception message.

Sounds like a hack but if you

  1. start an HTTP server in the background with a log endpoint (the endpoints simply writes everything on its console)
  2. patch the cy.log() command to make a request to your endpoint with the arguments it got

You should at least see the cy.log messages. It would be much more elegant if it goes through the existing cypress internal websocket though.

Anyone here tried something like this before?

@bahmutov

This comment has been minimized.

Collaborator

bahmutov commented May 7, 2018

For everyone who wants to get the cy.log and other Cypress commands this might be helpful https://github.com/bahmutov/cypress-failed-log

@jantoebes

This comment has been minimized.

jantoebes commented May 31, 2018

First, thanks for all the good work!

But I would address this particularly issue again.

To give a daily example. We couldnt reproduce a 'bug' in our application in local development and our CI server continuously failed. We were not able to find why, because we were not able to inspect the console errors (with an error when a network request failed). After a lot of research we discovered that our CI environment was not able to connect to a specific endpoint.

So you cannot guarantee that cypress test running locally will behave exact the same as on your CI.

It was a little bit frustrating, because in the summary of this ticket there was a line

We can pretty easily display console.log statements by mixing them into stdout when running headlessly

and that simple functionality would helped us so much.

As kind of a hack we now have a feature flag when running E2E on the CI that when having a console error, the page will output the error.

/* This part is to put console errors in the HTML when running cypress tests */
var console = (function(oldCons){
  return {
    log: function(text){
      oldCons.log(text);
    },
    info: function (text) {
      oldCons.info(text);
    },
    warn: function (text) {
      // $FlowFixMe
      document.getElementById('root').innerHTML = text
      oldCons.warn(text);
    },
    error: function (text) {
      // $FlowFixMe
      document.getElementById('root').innerHTML = text
      oldCons.error(text);
    }
  };
}(window.console));

if (window.envConfig.cypressConsole == 'true') {
  window.console = console;
}
@jennifer-shehane

This comment has been minimized.

Member

jennifer-shehane commented Jun 1, 2018

Since 3.0, we're collecting more data related to the timings, body, and hooks run for recorded tests.

This issue has been put on hold as there was a great deal of work to be done in the server to gather all of the data necessary. It is still on our roadmap for the future - and there has been some work done on it.

@souljacker

This comment has been minimized.

souljacker commented Jun 4, 2018

+1 for this being implemented

@CorayThan

This comment has been minimized.

CorayThan commented Aug 10, 2018

Yeah, if it's so easy to mix cy.log into the console output, then give us a CLI config option for outputting that to the console. Don't just leave us helpless for more than a year working on the perfect solution.

@zhex900

This comment has been minimized.

zhex900 commented Aug 28, 2018

Hi,
Is the dashboard logging tab still in the development plan?

@zachfeldman

This comment has been minimized.

zachfeldman commented Dec 3, 2018

We just had this issue, where tests pass locally, but fail on CI. We have no way of knowing why they fail as we just get a White Screen of Death and no console.log output. I'm at a loss for any way to know what error is being thrown as I can't even reproduce it locally. Any other tips for jury-rigging console logging on Travis?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment