Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--hide-successes is displaying test headings and names but not the result ... sometimes #152

Closed
jfischoff opened this issue Nov 14, 2016 · 20 comments

Comments

@jfischoff
Copy link

@jfischoff jfischoff commented Nov 14, 2016

We are using the --hide-successes flag and after adding 976 test it stopped functioning correctly. Instead of only displaying failures, it is displaying the heading and test names. We see this on both OSX and FreeBSD. Unfortunately I don't have a test case I can easily show just yet.

Setting TERM=dumb causes the correct output to display.

@feuerbach

This comment has been minimized.

Copy link
Owner

@feuerbach feuerbach commented Nov 14, 2016

Tasty uses ansi-terminal features to display the headings and the name of the current test while it's running, and then erase them if it hasn't failed. So it's possible that this may break under extreme conditions — perhaps when the amount of text to erase overflows the terminal or something.

It may be a while until I get around to looking into this myself, but patches are welcome.

@jfischoff

This comment has been minimized.

Copy link
Author

@jfischoff jfischoff commented Nov 14, 2016

Thanks that gives me something to go on

@feuerbach

This comment has been minimized.

Copy link
Owner

@feuerbach feuerbach commented Nov 29, 2016

Did you manage to create a simple reproducible example?

Just a deep test tree is not enough to trigger this. Here's what I tried:

import Test.Tasty
import Test.Tasty.HUnit
import Data.Function

deepTree :: TestTree
deepTree = flip fix depth $ \rec n ->
  if n == 0
    then testCase "Test case" $ return ()
    else testGroup ("Test group " ++ show (depth+1-n)) [rec (n-1)]
  where depth = 100

main = defaultMain deepTree
@feuerbach

This comment has been minimized.

Copy link
Owner

@feuerbach feuerbach commented Jun 13, 2017

Closing until there's some additional info.

@feuerbach feuerbach closed this Jun 13, 2017
@jfischoff

This comment has been minimized.

Copy link
Author

@jfischoff jfischoff commented Jun 13, 2017

👍

@ekmett-da

This comment has been minimized.

Copy link

@ekmett-da ekmett-da commented Apr 12, 2018

I can duplicate the result here at Digital Asset with our test harness. I get 800+ headers of various tests and then the bodies of the failing ones. This happens even when I set the terminal wide-enough that the text doesn't overflow. We do have tests layered several levels deep if that helps narrow the cause.

I also don't have a nice contained example, however.

@feuerbach

This comment has been minimized.

Copy link
Owner

@feuerbach feuerbach commented Apr 13, 2018

Would it be feasible for you to replace all test bodies with return () and see whether the problem persists? If it does, you could maybe share the test tree.

@ekmett-da

This comment has been minimized.

Copy link

@ekmett-da ekmett-da commented Apr 13, 2018

The tests are spread across probably 2-dozen different projects aggregated into one large test harness, making that a fair bit more awkward than it sounds.

@feuerbach feuerbach reopened this Apr 13, 2018
@feuerbach

This comment has been minimized.

Copy link
Owner

@feuerbach feuerbach commented Apr 13, 2018

This is under some sort of unix, right?

Could you do the following:

  1. Build tasty from the branch force-ansi
  2. Redirect the output of ./test --hide-successes to a file
  3. See if catting the file reproduces the problem
  4. If so, scrape the sensitive information and share the file
@luc-tielen

This comment has been minimized.

Copy link

@luc-tielen luc-tielen commented Oct 10, 2018

Hi,

I also came across this bug when working on a hobby project of mine.
I only have the error when I don't have my terminal full screen (for example only occupying right 50% of the screen).
Making it full screen makes the flag work again.
Also TERM=dumb seems to fix it also as mentioned above.

Some things I noticed:

  1. cat after redirecting to file works fine.
  2. OK does not appear in output (probably because it is hidden)
  3. FAIL appears in a weird location (probably due to wrap around of line?)
    Screenshot for clarification:
    image
    If you get this error, and then resize the width of the terminal, output gets weird too:
    image
    Kind of looks like it keeps thinking width stays the same. Newlines are messed up too.

NOTE: I did NOT run with tasty compiled on the force-ansi branch, don't know how to integrate that with stack (yet) :).
Hope this helps, it's an annoying bug and would really appreciate it being fixed!

Seems to be related to width of terminal? (Try sliding it back and forth around the width FAIL ends up completely on the next line).

@feuerbach

This comment has been minimized.

Copy link
Owner

@feuerbach feuerbach commented Oct 11, 2018

@luc-tielen could you post a reproducible example? Feel free to include only testGroups and trivial testCases that replicate your structure.

@luc-tielen

This comment has been minimized.

Copy link

@luc-tielen luc-tielen commented Oct 11, 2018

Sure, here you go: https://github.com/luc-tielen/hspec-terminal-bug

You can test it by running make test.
This can reproduce it for me when I change integer in the forM_ to a value >= 83 + change size of terminal to one half of my screen.
Full screen it does not occur.
Hope this helps, feel free to play around with example code.

@feuerbach

This comment has been minimized.

Copy link
Owner

@feuerbach feuerbach commented Oct 11, 2018

Yes, I can finally reproduce this — thanks!

@feuerbach

This comment has been minimized.

Copy link
Owner

@feuerbach feuerbach commented Oct 11, 2018

So the cause of the problem is that, when line wrapping occurs, the ANSI API treats the wrapped line as two separate lines. This is a bit surprising to me because if I resize the terminal back to the full size, the two lines become one — so the terminal remains aware that the two rendered lines are a single logical line.

Currently, tasty assumes that what is output as a single line really is a single line from ANSI point of view to be able to return on that line and to erase it.

We cannot use saveCursor/restoreCursor because they only support a single position, whereas we need an unbounded stack of positions.

We could use getCursorPosition, but it will be almost as fragile. If, while a long-running test is executing, you resize the terminal, then all saved coordinates become wrong.

@mpilgrem maybe you know a way to store a stack of terminal positions and be able to go back up to any of those positions that is robust to text wrapping and resizing the terminal?

@luc-tielen

This comment has been minimized.

Copy link

@luc-tielen luc-tielen commented Oct 13, 2018

I don't really know the implementation of the code, but are all results printed at end? Or as they come in?
If they all arrive at the end, why not filter out the output from successful tests if the flag is set?
It might even be possible as they come in..

Might be a lot easier than ANSI terminal magic.. something worth looking into?

@mpilgrem

This comment has been minimized.

Copy link

@mpilgrem mpilgrem commented Oct 13, 2018

Nothing occurs to me so far. For Windows users, I found this blog entry that discusses under 'word wrap' how the native Windows console handles things (behind the scenes, 'this line has been wrapped here' codes are recorded in the screen buffer). In those consoles, 'Wrap text output on resize' is a layout option that a user can disable/enable and - I think, based on some searches - that the API does not expose what the user has chosen. I think that alone makes it near impossible to have an approach that is robust to terminal resizing.

@feuerbach

This comment has been minimized.

Copy link
Owner

@feuerbach feuerbach commented Oct 13, 2018

@luc-tielen the results are printed as they come in, and then the successful ones are erased. This way you get immediate feedback about which test is executing (especially if it's taking too long). Here's how it looks.

If you don't need this immediate feedback, then exporting TERM=dumb seems like a decent solution — it will tell tasty that if though this looks like a termnal, tasty shouldn't attempt to use its ANSI features.

@mpilgrem thanks for looking into this!

@feuerbach

This comment has been minimized.

Copy link
Owner

@feuerbach feuerbach commented Oct 13, 2018

Hm, the issue with TERM=dumb is that it also disables color. I should add a flag to only disable erasing.

@gromakovsky

This comment has been minimized.

Copy link

@gromakovsky gromakovsky commented Apr 29, 2019

For me --hide-successes also doesn't work when hSupportsANSI stdout returns True. I've found this issue and it looks like now it's broken even more than it was before. Apparently consoleOutputHidingSuccesses erases less than it should. According to the comments above the issues happens only if terminal is not wide enough, but for me it now happens even in wide terminal.

Here is a tiny repo which reproduces the problem.

  1. If I run this test with TERM=dumb, it doesn't print any names (neither group names nor test case names). This is the behavior I expect and I assume that without TERM=dumb it should behave the same way.
  2. If I run it with --hide-successes in wide terminal, I will see only group names:
Tests
  q
  1. If I run it with --hide-successes in narrow terminal, I will see names of individual tests as well:
Tests
  2+2=4:                                                                                                                                                       8 is even                                                                                                                                                    q
    3+3=6:  

Note that 8 is even test has very long name (see sources), so in narrow terminal it occupies two lines and the last line is erased.

I understand that dealing with narrow terminals is hard for the reasons described above in this thread, but --hide-successes seems to be broken for wide terminals as well and I guess it might be easier to fix.

feuerbach added a commit that referenced this issue May 11, 2019
@feuerbach

This comment has been minimized.

Copy link
Owner

@feuerbach feuerbach commented May 11, 2019

All: there is now a fix/workaround in master: you can pass --ansi-tricks=false on the command line or set TASTY_ANSI_TRICKS=false in the environment to disable the ANSI tricks. Could you please try it and confirm if it works for you?

feuerbach added a commit that referenced this issue May 12, 2019
@feuerbach feuerbach closed this Jul 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.