Strictness in Mockito #769

Open
szczepiq opened this Issue Nov 19, 2016 · 5 comments

Projects

None yet

4 participants

@szczepiq
Member
szczepiq commented Nov 19, 2016 edited

Introduction

This issue explores the addition of "strict" behavior and APIs to Mockito. Traditionally, Mockito is a lenient mocking framework. For background and motivation, check out Szczepan's article on LinkedIn.

Why strictness in Mockito:

  • productivity - improved debugging and writing tests. Currently MockitoHint warns about stubbing argument mismatches. Strict stubbing would promote the warning into an exception, making the test fail fast and improve debuggability / productivity.
  • clean code - avoid unnecessary stubbings, test fails when unncessary stubs are present.

Concrete API proposals:

  • "strict stubbing" proposal: #770
  • "strict mocking" proposal: #649

Example exception message indicating subbing problem:

org.mockito.exceptions.misusing.PotentialStubbingProblem:
Strict stubbing argument mismatch. Please check:
- this invocation of 'simpleMethod' method:
    mock.simpleMethod(“Foo");
      -> at … StrictStubbingEndToEndTest.java:101
- has following stubbing(s) with different arguments:
    1. mock.simpleMethod(“foo");
      -> at … StrictStubbingEndToEndTest.java:100
Typically, stubbing argument mismatch indicates user mistake when writing tests.
Mockito fails early so that you can debug potential problem easily.

Details

Strictness in Mockito can have 2 forms:

  • Strict stubbing that requires that all declared stubs are actually used
  • Strict mocks, that require all non-void methods to be stubbed before they are called

Strictness behavior can be implemented as:

  • warning on the console
  • exception thrown

Strictness API can be implemented in:

  • JUnit runner / rule
  • TestNG runner
  • explicit method to call by the user

Future direction:

  • strict stubbing on by default, opt-out available
  • strict mocking off by default, opt-in available

Implementation as of Mockito 2.2.x:

  • JUnitRunner - fails at the end of test class when unused stubbings are present. Opt out available via JUnitRunner.Silent
  • JUnitRule - warn at the end of test method about a) unused stubs b) stubs invoked with different arguments. Opt-out available via rule().silent().

Mockito 2.x proposal:

  1. New "strict stubbing" API for JUnit rule - #770
  2. New Answer.StrictMock that fails immediately when non-void method invoked on without prior stubbing - #649
  3. new verifyStubs() method to use when rule/runner not in used

Mockito 3 proposal:

  • strict stubbing is the default
  • change behavior of verifyNoMoreInvocations()
  • runner and rule behave the same
@szczepiq szczepiq added a commit that referenced this issue Nov 19, 2016
@szczepiq szczepiq Refactoring JUnit rule implementation
Needed for work on Mockito strictness, see #769. Decoupled JUnit API from the work Mockito does before and after every test.

No public API exposed, listeners and friends need to be refactored and documented.

Don't merge. This refactoring on its own does not add value.
35c03ab
@imod
imod commented Nov 21, 2016

I think from a code maintenance point of view, "behavior: warning on the console" is not really useful - in such a case a CI system would not inform anyone about unused stubs. An exception would be a lot better.

@szczepiq szczepiq added a commit that referenced this issue Dec 10, 2016
@szczepiq szczepiq Refactoring JUnit rule implementation
Needed for work on Mockito strictness, see #769. Decoupled JUnit API from the work Mockito does before and after every test.

No public API exposed, listeners and friends need to be refactored and documented.

Don't merge. This refactoring on its own does not add value.
53ea392
@szczepiq szczepiq added a commit that referenced this issue Dec 11, 2016
@szczepiq szczepiq Refactoring JUnit rule implementation
Needed for work on Mockito strictness, see #769. Decoupled JUnit API from the work Mockito does before and after every test.

No public API exposed, listeners and friends need to be refactored and documented.

Don't merge. This refactoring on its own does not add value.
2de2cba
@TimvdLippe
Contributor

@szczepiq could you add list items (- [ ] in markdown) to signify what the remaining action points in this issue are? That gives a clearer overview, especially as multiple open issues are related to this single epic.

@TimvdLippe TimvdLippe added the epic label Dec 13, 2016
@szczepiq szczepiq added the strictness label Dec 30, 2016
@szczepiq
Member

@szczepiq could you add list items (- [ ] in markdown) to signify what the remaining action points in this issue are? That gives a clearer overview, especially as multiple open issues are related to this single epic.

I will do that once this epic turns into an execution plan. Currently it is intended to scope out the details and I don't know if all of them will be executed (and in what form). I annotated one item as merged by using strikethrough. I hope it is clearer now.

@micheljung
micheljung commented Jan 31, 2017 edited

Since your Javadoc asks for feedback, here's my case:

  @Test
  public void testAutoCompleteDoesntCompleteWhenTheresNoWordBeforeCaret() throws Exception {
    when(playerService.getPlayerNames()).thenReturn(FXCollections.observableSet("DummyUser", "Junit"));
    textInputControl.setText("j");
    textInputControl.positionCaret(0);

    simulate(keyEvent(KeyCode.TAB));

    assertThat(textInputControl.getText(), is("j"));
  }

It tests auto-completion of an input field.

getPlayerNames() is stubbed, but never called in working code - because if the caret is at position 0, there should be no autocompletion. If I remove the stubbing it's possible to break the autocompletion without being detected; the test would succeed not because it's properly implemented, but because there are no users being returned.

Update:
After hitting enter I realized that I can improve this using verify(playerService, never()).getPlayerNames().
I decided to not remove the comment though :)

@szczepiq
Member
szczepiq commented Feb 1, 2017

@micheljung, my recommendation for this scenario is exactly using verify + never :) I will assume that your use case is satisfied. Looks like strict stubbing nudged you to improve this test!

Thank you very much for reporting. This is exactly kind of feedback we're looking for when evaluating strictness feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment