Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Counter examples missing from failing tests #36

Open
frekw opened this issue Jan 10, 2017 · 13 comments

Comments

Projects
None yet
8 participants
@frekw
Copy link

commented Jan 10, 2017

Hi!

When running tests that fail, ExCheck fails to print the failing counter example (which would be the expected behaviour, I assume).

For example, changing the sample to this:

  def prop_push_list do
    for_all({x, y}, {int(), list(int())}) do
      result = push(x, y)
      Enum.at(result, 0) == x and Enum.count(result) == Enum.count(y)
    end
  end

which is expected to fail, the output is the following:

ExCheck.SampleTest
  * test verify sample property (24.8ms)...........................................................................................
..........
  1) test verify sample property (ExCheck.SampleTest)
     test/sample_test.exs:5
     Expected truthy, got false
     code: ExCheck.check(ExCheck.Sample)
     stacktrace:
       test/sample_test.exs:6: (test)

  * test verify sample property with iteration parameter (20.5ms)

  2) test verify sample property with iteration parameter (ExCheck.SampleTest)
     test/sample_test.exs:9
     Expected truthy, got false
     code: ExCheck.check(ExCheck.Sample, 10)
     stacktrace:
       test/sample_test.exs:10: (test)

which doesn't provide me with much help for actually tracking down why a test is failing. Is this expected behaviour?

I'm running Elixir 1.4.0.

@avocade

This comment has been minimized.

Copy link

commented Jan 18, 2017

👍

@pdawczak

This comment has been minimized.

Copy link

commented Mar 23, 2017

I can confirm I have similar problem. The test fails as expected, but there is no hint regarding the problem:

screen shot 2017-03-23 at 20 20 48

@sotojuan

This comment has been minimized.

Copy link

commented Apr 15, 2017

Can anything be done @parroty? This can really make debugging tests hard.

@aerosol

This comment has been minimized.

Copy link

commented Jul 3, 2017

FWIW #37 solves it.

BTW I'll work on getting https://github.com/triqng/triq/pull/28 merged so that no IO capturing will be necessary in excheck.

@zkessin

This comment has been minimized.

Copy link

commented Jul 3, 2017

I merged triqng/triq#28

@aerosol

This comment has been minimized.

Copy link

commented Jul 3, 2017

Submitted first pass on #41

@parroty

This comment has been minimized.

Copy link
Owner

commented Jul 3, 2017

Sorry about late responses, and thanks for the PRs.

@aerosol: Could you help me understanding the relationship between #37 and #41? (Should I merge both? or just #41?).

@aerosol

This comment has been minimized.

Copy link

commented Jul 3, 2017

@parroty I pushed #37 to my branch and built stuff on top of it, effectively removing the module modified in #37. I could rewrite the commit history in #41, it has some rough at edges I've been meaning to take care of anyway. Sounds like a plan?

@aerosol

This comment has been minimized.

Copy link

commented Jul 3, 2017

cc @obrok

@parroty

This comment has been minimized.

Copy link
Owner

commented Jul 3, 2017

That sounds good (I understood that just merging #41 would be enough, and as long as original issue can be resolved, rewriting the history may not be required).

I can see one comment fixme. no counter example here in the PR. Is it ready? (or are you still working on this item?)

@obrok

This comment has been minimized.

Copy link

commented Jul 3, 2017

@parroty please just close #37 when #41 is merged

@aerosol

This comment has been minimized.

Copy link

commented Jul 3, 2017

Thanks @obrok

I can see one comment fixme. no counter example here in the PR. Is it ready? (or are you still working on this item?

@parroty Yes, I'd like to work on this PR a bit more. If you're OK with the general idea, I'll finish up a few things and ping you for a merge.

@parroty

This comment has been minimized.

Copy link
Owner

commented Jul 3, 2017

Great, thanks @aerosol @obrok.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.