JUnit rule logs warnings about unsued / misused stubs #384

Closed
mockitoguy opened this Issue Apr 7, 2016 · 10 comments

Comments

Projects
None yet
6 participants
@mockitoguy
Member

mockitoguy commented Apr 7, 2016

Why

When test fails, the failure might be caused by misused stubs. Hence, it might be worth to log out debugging information to the System out (e.g. misused stubs, unused stubs). For more details see documentation for MockitoHint.

Plan

  • document 2.x change
  • tweak the warning message
  • add behavior to the runner
  • add silent runner
  • add silent setting to the rule (or warnings level for all / exception only / none)
  • deprecate/remove console spamming runner

Impl

The JUnit rule or the runner will potentially include following info in the output (somehow): https://gist.github.com/szczepiq/38619fd8766c66c46dff4dc2b6db06ec

I'm wondering if "Mockito.validateMockitoUsage();" should automatically print warnings. We could provide boolean parameter to control printing of the warnings.

@mockitoguy mockitoguy added this to the 2.0 milestone Apr 7, 2016

@mockitoguy mockitoguy changed the title from Make JUnit rule more useful to JUnit rule on test failure logs warnings about unsued / misused stubs Apr 13, 2016

mockitoguy added a commit that referenced this issue Apr 13, 2016

Mockito JUnit Rule logs warnings
See issue #384

When test fails, it is useful to know about unused / misused stubs.

mockitoguy added a commit that referenced this issue Apr 18, 2016

mockitoguy added a commit that referenced this issue Apr 18, 2016

Tweaked the output produced by JUnitRule
Making the output look better in real tests.

See issue #384

mockitoguy added a commit that referenced this issue Apr 18, 2016

Tweaked the output produced by JUnitRule
Tweaked the output and improve test coverage (the case of multiple different kinds of stubbing)

See issue #384

mockitoguy added a commit that referenced this issue Apr 18, 2016

Tweaked the output produced by JUnitRule
Tweaked the output again.

See issue #384

mockitoguy added a commit that referenced this issue Apr 18, 2016

Tightened integ test coverage
Top level test now asserts on full content of the stubbing information. Added handy method for making assertions stable and the test easier to work with.

See issue #384

@TimvdLippe TimvdLippe added the ready label Jul 12, 2016

@TimvdLippe TimvdLippe modified the milestones: 2.1, 2.0 Jul 13, 2016

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Jul 24, 2016

Member

On it.

Member

mockitoguy commented Jul 24, 2016

On it.

@mockitoguy mockitoguy self-assigned this Jul 24, 2016

mockitoguy added a commit that referenced this issue Jul 31, 2016

Started designing the code changes for issue #384
I'm planning to rework the current behavior of JUnit rules, so that the rule by default:
 - on test failure print the stubbing mismatch
 - when test passes show unused stubs

It will use the new StubbingListener interface. WarningsCollector class and friends will be deleted.

mockitoguy added a commit that referenced this issue Jul 31, 2016

Working my way to fix issue #384
Added a new style of events that the StubbingListener can listen to.
I need information when the mock invocation was triggered but no stubbed behavior was registered for this invocation.

mockitoguy added a commit that referenced this issue Jul 31, 2016

mockitoguy added a commit that referenced this issue Jul 31, 2016

mockitoguy added a commit that referenced this issue Jul 31, 2016

mockitoguy added a commit that referenced this issue Jul 31, 2016

Moved class to test source tree
This class was only used in tests, so it's better if it lives in test source tree.

Issue #384

mockitoguy added a commit that referenced this issue Jul 31, 2016

Cleaned up the design
Removed unnecessary conditional logic

Issue #384

mockitoguy added a commit that referenced this issue Jul 31, 2016

mockitoguy added a commit that referenced this issue Jul 31, 2016

mockitoguy added a commit that referenced this issue Jul 31, 2016

mockitoguy added a commit that referenced this issue Jul 31, 2016

Refactoring
Busted bigger class into something smaller

Issue #384

mockitoguy added a commit that referenced this issue Jul 31, 2016

Improved the reporter
Ensured the stubs are reported in the order they are declared. This is not unit testable cleanly.

Issue #384

mockitoguy added a commit that referenced this issue Jul 31, 2016

Improved the JUnit rule stubbing warnings reporting
Reporting needs to be based on the actual invocation, not based on String representation of the location.

Issue #384

mockitoguy added a commit that referenced this issue Jul 31, 2016

Improved the JUnit rule stubbing warnings reporting
- more cases covered
- more coverage

Issue #384

mockitoguy added a commit that referenced this issue Jul 31, 2016

mockitoguy added a commit that referenced this issue Jul 31, 2016

Updated TODO
Issue #384

@TimvdLippe TimvdLippe added in progress and removed ready labels Jul 31, 2016

mockitoguy added a commit that referenced this issue Jul 31, 2016

Ensured that Rule can coexists with Runner
I don't have strong opinion what should be the desired behavior in this case. I just want this scenario be reasonably graceful. Added test to cover this.

Issue #384

mockitoguy added a commit that referenced this issue Jul 31, 2016

mockitoguy added a commit that referenced this issue Jul 31, 2016

Fixed the code
Thank you, tests.

Issue #384

mockitoguy added a commit that referenced this issue Jul 31, 2016

mockitoguy added a commit that referenced this issue Aug 9, 2016

mockitoguy added a commit that referenced this issue Aug 9, 2016

mockitoguy added a commit that referenced this issue Aug 9, 2016

Unexposed APIs that don't have to be public
Not ready yet for MockitoDebugger

#384

mockitoguy added a commit that referenced this issue Aug 9, 2016

mockitoguy added a commit that referenced this issue Aug 9, 2016

Started using a collection of listeners
Followed by great feedback from Tim

#384

mockitoguy added a commit that referenced this issue Aug 9, 2016

mockitoguy added a commit that referenced this issue Aug 9, 2016

mockitoguy added a commit that referenced this issue Aug 9, 2016

mockitoguy added a commit that referenced this issue Aug 9, 2016

mockitoguy added a commit that referenced this issue Aug 9, 2016

mockitoguy added a commit that referenced this issue Aug 9, 2016

@mockitoguy mockitoguy changed the title from JUnit rule on test failure logs warnings about unsued / misused stubs to JUnit rule logs warnings about unsued / misused stubs Aug 11, 2016

mockitoguy added a commit that referenced this issue Aug 11, 2016

mockitoguy added a commit that referenced this issue Aug 12, 2016

mockitoguy added a commit that referenced this issue Aug 13, 2016

Rename job for clarity
Following great feedback from Christian.

Issue #384

mockitoguy added a commit that referenced this issue Aug 13, 2016

Protected from null argument
Following neat feedback from Christian.

Issue #384

@TimvdLippe TimvdLippe modified the milestones: 2.0, 2.1 Aug 14, 2016

mockitoguy added a commit that referenced this issue Aug 14, 2016

Removed unnecessary cast
Following neat feedback from Brice.

Issue #384

mockitoguy added a commit that referenced this issue Aug 14, 2016

mockitoguy added a commit that referenced this issue Aug 14, 2016

mockitoguy added a commit that referenced this issue Aug 14, 2016

JUnit rules report unused stubs
- this commit squashes 50+ commits
- introduces new MockitoListener API

Fixed issue #384

@TimvdLippe TimvdLippe removed the in progress label Aug 16, 2016

mockitoguy added a commit that referenced this issue Aug 19, 2016

Made the JUnit runner thread safe
 - unused stubbing detection logic was not thread safe in Mockito JUnit runner
 - stopped using the StubbingListener - by design it couldn't support thread safe scenarios unfortunately :(

Fixes #384

mockitoguy added a commit that referenced this issue Aug 19, 2016

Made the JUnit runner thread safe
 - unused stubbing detection logic was not thread safe in Mockito JUnit runner
 - stopped using the StubbingListener - by design it couldn't support thread safe scenarios unfortunately :(

Fixes #384
@t1

This comment has been minimized.

Show comment
Hide comment
@t1

t1 Aug 29, 2016

@szczepiq: I like this feature very much. Also that the MockitoJUnitRunner fails by default is good. I think a note in the javadoc of UnnecessaryStubbingException to use MockitoJUnitRunner.Silent would be helpful.

We often have quite complex test fixtures, and it's cumbersome to switch absolutely each and every stub on or off... some methods simply have to be there; it's also not necessary to verify if they've been invoked. So it would be great to be able to disable it for some stubs, but leave it on for others.

t1 commented Aug 29, 2016

@szczepiq: I like this feature very much. Also that the MockitoJUnitRunner fails by default is good. I think a note in the javadoc of UnnecessaryStubbingException to use MockitoJUnitRunner.Silent would be helpful.

We often have quite complex test fixtures, and it's cumbersome to switch absolutely each and every stub on or off... some methods simply have to be there; it's also not necessary to verify if they've been invoked. So it would be great to be able to disable it for some stubs, but leave it on for others.

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Aug 29, 2016

Contributor

@t1 That seems like a reasonable idea. Want to make a PR for that?

Contributor

TimvdLippe commented Aug 29, 2016

@t1 That seems like a reasonable idea. Want to make a PR for that?

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Aug 29, 2016

Member

Good feedback. I think that MockitoHint class should also mention that it is possible to silence the new mechanism.

Member

mockitoguy commented Aug 29, 2016

Good feedback. I think that MockitoHint class should also mention that it is possible to silence the new mechanism.

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
Member

mockitoguy commented Aug 29, 2016

@Sarev0k

This comment has been minimized.

Show comment
Hide comment
@Sarev0k

Sarev0k Oct 26, 2016

The MockitoHint documentation seems to indicate that this will give you warnings about mismatched arguments, when using the MockitoJUnitRunner. However, from my experience this doesn't seem to work with Mockito 2.1.

Here's the source I'm trying:

Foo foo = mock(Foo.class);
when(foo.bar("baz")).thenReturn("bar");

foo.bar("baz");
foo.bar("test"); // expected to get a warning here

Is there something else I need to do to enable this functionality?

Sarev0k commented Oct 26, 2016

The MockitoHint documentation seems to indicate that this will give you warnings about mismatched arguments, when using the MockitoJUnitRunner. However, from my experience this doesn't seem to work with Mockito 2.1.

Here's the source I'm trying:

Foo foo = mock(Foo.class);
when(foo.bar("baz")).thenReturn("bar");

foo.bar("baz");
foo.bar("test"); // expected to get a warning here

Is there something else I need to do to enable this functionality?

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Nov 1, 2016

Member

Thank you for feedback!

Is there something else I need to do to enable this functionality?

This is how they were implemented. Runner should also report mismatches because it helps with debugging. The reason it does not do it today is because I wanted to limit the noise.

We will fix this issue. Also, we will consider always printing warning when stub args mismatch, even if one does not use runner / rule.

Do you want to open a separate ticket to track this improvement?

Thanks for reporting!

Member

mockitoguy commented Nov 1, 2016

Thank you for feedback!

Is there something else I need to do to enable this functionality?

This is how they were implemented. Runner should also report mismatches because it helps with debugging. The reason it does not do it today is because I wanted to limit the noise.

We will fix this issue. Also, we will consider always printing warning when stub args mismatch, even if one does not use runner / rule.

Do you want to open a separate ticket to track this improvement?

Thanks for reporting!

@Sarev0k

This comment has been minimized.

Show comment
Hide comment
@Sarev0k

Sarev0k Nov 1, 2016

I've opened #725 to track this.

Sarev0k commented Nov 1, 2016

I've opened #725 to track this.

@mockitoguy mockitoguy referenced this issue Jan 18, 2017

Merged

New strict stubbing API - MockitoSession #865

19 of 19 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment