Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strictness in Mockito #769

Open
mockitoguy opened this issue Nov 19, 2016 · 54 comments
Open

Strictness in Mockito #769

mockitoguy opened this issue Nov 19, 2016 · 54 comments

Comments

@mockitoguy
Copy link
Member

mockitoguy commented Nov 19, 2016

Introduction

This issue explores the addition of "strict" behavior and APIs to Mockito. Traditionally, Mockito is a lenient mocking framework. For background and motivation, check out Szczepan's short article on LinkedIn or in-depth article on Mockito blog.

Why strictness in Mockito:

  • productivity - improved debugging and writing tests. Currently MockitoHint warns about stubbing argument mismatches. Strict stubbing would promote the warning into an exception, making the test fail fast and improve debuggability / productivity.
  • clean code - avoid unnecessary stubbings, test fails when unncessary stubs are present.

Concrete API proposals:

Example exception message indicating subbing problem:

org.mockito.exceptions.misusing.PotentialStubbingProblem:
Strict stubbing argument mismatch. Please check:
- this invocation of 'simpleMethod' method:
    mock.simpleMethod(“Foo");
      -> at … StrictStubbingEndToEndTest.java:101
- has following stubbing(s) with different arguments:
    1. mock.simpleMethod(“foo");
      -> at … StrictStubbingEndToEndTest.java:100
Typically, stubbing argument mismatch indicates user mistake when writing tests.
Mockito fails early so that you can debug potential problem easily.

Details

Strictness in Mockito can have 2 forms:

  • Strict stubbing that requires that all declared stubs are actually used
  • Strict mocks, that require all non-void methods to be stubbed before they are called

Strictness behavior can be implemented as:

  • warning on the console
  • exception thrown

Strictness API can be implemented in:

  • JUnit runner / rule
  • TestNG runner
  • explicit method to call by the user

Future direction:

  • strict stubbing on by default, opt-out available
  • strict mocking off by default, opt-in available

Implementation as of Mockito 2.2.x:

  • JUnitRunner - fails at the end of test class when unused stubbings are present. Opt out available via JUnitRunner.Silent
  • JUnitRule - warn at the end of test method about a) unused stubs b) stubs invoked with different arguments. Opt-out available via rule().silent().

Mockito 2.x proposal:

  1. New "strict stubbing" API for JUnit rule - Opt-in stubbing strictness implemented in JUnit rules #770
  2. New Answer.StrictMock that fails immediately when non-void method invoked on without prior stubbing - Better support of strict mocks #649
  3. new verifyStubs() method to use when rule/runner not in used

Mockito 3 proposal:

  • strict stubbing is the default
  • change behavior of verifyNoMoreInvocations()
  • runner and rule behave the same
mockitoguy added a commit that referenced this issue Nov 19, 2016
Needed for work on Mockito strictness, see #769. Decoupled JUnit API from the work Mockito does before and after every test.

No public API exposed, listeners and friends need to be refactored and documented.

Don't merge. This refactoring on its own does not add value.
@imod
Copy link

imod commented Nov 21, 2016

I think from a code maintenance point of view, "behavior: warning on the console" is not really useful - in such a case a CI system would not inform anyone about unused stubs. An exception would be a lot better.

mockitoguy added a commit that referenced this issue Dec 10, 2016
Needed for work on Mockito strictness, see #769. Decoupled JUnit API from the work Mockito does before and after every test.

No public API exposed, listeners and friends need to be refactored and documented.

Don't merge. This refactoring on its own does not add value.
mockitoguy added a commit that referenced this issue Dec 11, 2016
Needed for work on Mockito strictness, see #769. Decoupled JUnit API from the work Mockito does before and after every test.

No public API exposed, listeners and friends need to be refactored and documented.

Don't merge. This refactoring on its own does not add value.
@TimvdLippe
Copy link
Contributor

TimvdLippe commented Dec 13, 2016

@szczepiq could you add list items (- [ ] in markdown) to signify what the remaining action points in this issue are? That gives a clearer overview, especially as multiple open issues are related to this single epic.

@mockitoguy
Copy link
Member Author

mockitoguy commented Jan 13, 2017

@szczepiq could you add list items (- [ ] in markdown) to signify what the remaining action points in this issue are? That gives a clearer overview, especially as multiple open issues are related to this single epic.

I will do that once this epic turns into an execution plan. Currently it is intended to scope out the details and I don't know if all of them will be executed (and in what form). I annotated one item as merged by using strikethrough. I hope it is clearer now.

@micheljung
Copy link

micheljung commented Jan 31, 2017

Since your Javadoc asks for feedback, here's my case:

  @Test
  public void testAutoCompleteDoesntCompleteWhenTheresNoWordBeforeCaret() throws Exception {
    when(playerService.getPlayerNames()).thenReturn(FXCollections.observableSet("DummyUser", "Junit"));
    textInputControl.setText("j");
    textInputControl.positionCaret(0);

    simulate(keyEvent(KeyCode.TAB));

    assertThat(textInputControl.getText(), is("j"));
  }

It tests auto-completion of an input field.

getPlayerNames() is stubbed, but never called in working code - because if the caret is at position 0, there should be no autocompletion. If I remove the stubbing it's possible to break the autocompletion without being detected; the test would succeed not because it's properly implemented, but because there are no users being returned.

Update:
After hitting enter I realized that I can improve this using verify(playerService, never()).getPlayerNames().
I decided to not remove the comment though :)

@mockitoguy
Copy link
Member Author

mockitoguy commented Feb 1, 2017

@micheljung, my recommendation for this scenario is exactly using verify + never :) I will assume that your use case is satisfied. Looks like strict stubbing nudged you to improve this test!

Thank you very much for reporting. This is exactly kind of feedback we're looking for when evaluating strictness feature.

@skestle
Copy link

skestle commented Jul 21, 2017

I would like an option to verifyNoMoreInteractions in a smart way:

  • ignoring expected/stub invocations as in STRICT_STUBS mode
  • ignoring unnecessary stubs methods as in non-strict modes.

I'm simulating an external API (Selenium) to ensure I wrap it properly.

	public void expectToFindNoElement(final By by) {
		when(this.webElement.findElement(by)).thenThrow(new NoSuchElementException("Cannot locate an element " + by));
		when(this.webElement.findElements(by)).thenReturn(Collections.emptyList());
	}

It's very easy to accidentally not throw the exception when we're expecting no element; the correct thing selenium consumers generally want to do is verify no elements.

Essentially, I can't be DRY with Strict without breaking it's strict "clean and maintainable" expectations.

BTW, I agree this is not "clean" but it is certainly very much more maintainable (and correct) than what STRICT mode allows.

@skestle
Copy link

skestle commented Jul 21, 2017

So in terms of 8.5.47's implementation, I'd like to haveUniversalTestListener.testFinished in LENIENT mode, while DefaultStubbingLookupListener.onStubbingLookup executes invocation.markVerified(), as it does in strict mode.

Probably what I want is DRY without strictness as mockito wants to define it. IMO, DefaultStubbingLookupListener and DRY verifyNoMoreInteractions() has nothing to do with strictness.
Please separate these two concepts.

@skestle
Copy link

skestle commented Jul 21, 2017

Also, having tried for half an hour trying to grok StubbingLookupListeners, CreationSettings MockitoTestListeners etc, it seems there's no way to insert invocation.markVerified() without creating my own Test Runner.

If there's some sort of workaround where I can make this auto-verification automatic, please respond. Should there be another ticket to add auto-verification of stubbed invocations (which is an awesome feature BTW) that's separate from this ticket (as it should be)?

@mockitoguy
Copy link
Member Author

mockitoguy commented Jul 21, 2017

Thank you for reporting and debugging Mockito strictness! Let me give you some feedback about the use case first:

I would like an option to verifyNoMoreInteractions in a smart way:
ignoring expected/stub invocations as in STRICT_STUBS mode
ignoring unnecessary stubs methods as in non-strict modes.

It seems that you are suggesting a different Strictness level, something between "lenient" and "strict stubs":

  • lenient - allow unused stubbings, veryNoMoreInteractions catches all
  • "kinda strict stubs" - allow unused stubbings, verifyNoMoreInteractions ignores used stubbings
  • strict stubs - disallow unused stubbings, verifyNoMoreInteractions ignores used stubbings

Have you tried this:

verifyNoMoreInteractions(ignoreStubs(mock1, mock2));

Before we jump to the impl details of TestListener (which is an internal type, not intended for regular use, but absolutely open to experimentation and creative usage), let's nail down the use case first.

Thank you for reporting! This is a very interesting use case.

@karollewandowski
Copy link

karollewandowski commented Aug 22, 2017

I like strict stubbing very much.
I've seen opinion that it allows to remove a lot of code, but I think that it rather forces to write a little more. I used to put common stubbing code to setup method even if it wasn't used by all tests. Now I have to move it to tests actually using it. I think that it's nothing bad and makes my tests clearer. When stubbing code is in test method, then there is no need to scroll source file to top to check what is stubbed and what is not. Everything is in one place and test context is explicit and close to it's usage. More code doesn't mean duplications - common stubbing can be extracted to method with descriptive name.
@szczepiq, on Twitter you mentioned that you are wondering API like when(method, optional()).then(...). I think that it's not necessary, but sometimes can be helpful.
It would allow to create optional stubbing in setup method what would break clarity provided by strict stubbing again. On the other hand, sometimes we don't care about some dependency or there are tens of tests using some stubbing and one not, so optional() would be nice to handle such cases (and it clearly says it's optional).

@adamvoss
Copy link

adamvoss commented Jan 30, 2018

Can this be exposed by MockSettings or otherwise be made available without requiring the Mock instance be a class member.

The following "works" but uses a whole lot more internal stuff than I like:

import org.mockito.MockSettings;
import org.mockito.Mockito;
import org.mockito.internal.creation.settings.CreationSettings;
import org.mockito.internal.listeners.StubbingLookupListener;
import org.mockito.mock.MockCreationSettings;
import org.mockito.quality.Strictness;

import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;

import static org.mockito.Mockito.withSettings;

public final class StrictMockito
{
   private StrictMockito()
   {
   }

   public static <T> T strictMock(Class<T> classToMock) {
      final MockSettings mockSettings = withSettings();
      createDefaultStubbingLookupListener(Strictness.STRICT_STUBS);
      final T mock = Mockito.mock(classToMock, mockSettings);
      final MockCreationSettings<?> mockCreationSettings = Mockito.mockingDetails(mock).getMockCreationSettings();
      ((CreationSettings<?>)mockCreationSettings).getStubbingLookupListeners().add( createDefaultStubbingLookupListener(Strictness.STRICT_STUBS));
      return mock;
   }

   private static StubbingLookupListener createDefaultStubbingLookupListener(Strictness strictness)
   {
      try
      {
         final Class<?> clazz = Class.forName("org.mockito.internal.junit.DefaultStubbingLookupListener");
         final Constructor<?> constructor = clazz.getDeclaredConstructor(Strictness.class);
         constructor.setAccessible(true);
         return (StubbingLookupListener) constructor.newInstance(strictness);
      }
      catch (ClassNotFoundException | NoSuchMethodException | IllegalAccessException | InstantiationException | InvocationTargetException e)
      {
         throw new RuntimeException(e);
      }
   }
}

@adamvoss
Copy link

adamvoss commented Jan 30, 2018

If possible, the failure message around passing null and collections could use improvement. As is, we can get errors like the following:

Strict stubbing argument mismatch. Please check:
 - this invocation of 'getValue' method:
    sut.getValue("", null, []);
    -> at ExampleTest.strictExample(ExampleTest.java:64)
 - has following stubbing(s) with different arguments:
    1. sut.getValue("", null, null);
      -> at ExampleTest.strictExample(ExampleTest.java:62)

It is not clear from the failure message that only the middle argument is the source of the problem. One is likely to expect the final argument is the problem because it is the entry that is different.

Code to reproduce the above:

   public interface Sut
   {
      Object getValue(String foo, Object bar, Set<String> baz);
   }

   @Mock
   Sut sut;

   @Test
   public void strictExample() {
      Mockito.when(sut.getValue(ArgumentMatchers.anyString(), ArgumentMatchers.any(Object.class),
            ArgumentMatchers.eq(Collections.emptySet()))).thenReturn(null);

      sut.getValue("", null, Collections.emptySet());
   }

@mockitoguy
Copy link
Member Author

mockitoguy commented Jan 31, 2018

Hey @adamvoss! Thanks for checking in!

Can this be exposed by MockSettings or otherwise be made available without requiring the Mock instance be a class member

Mocks don't have to be class members to leverage strictness. You can use MockitoSession object to start mocking before you create mocks. Would that solve your problem?

@adamvoss
Copy link

adamvoss commented Feb 16, 2018

@mockitoguy Sorry, I missed the notification on this one.

Here is what that looks like:

public final class MockTest
{
   @Test
   public void doTest() {
      final MockitoSession session = Mockito.mockitoSession().strictness(Strictness.STRICT_STUBS).startMocking();
      final IWhatever mock = Mockito.mock(IWhatever.class);
      when(mock.getResult("bar")).thenReturn("yes");
      Assert.assertNotNull(mock.getResult("foo"));
      
      // Should be in tear down (after) method
      session.finishMocking();
   }
}


interface IWhatever {
   public String getResult(String foo);
}

I would say that is more flexible, but not exactly what I was looking for. One of the motivations being able to specify it on a per-mock setting is to be able to introduce them gradually. That way I can use strict when I want to (including existing tests) without changing any of the existing mocks.

With you wanting to make this eventually the default mode, then I can see the argument for not allowing granular configuration. (Though once on v3, people may want to gradually use some lenient mocks). That aside, it is an annoyance to need to setup and tear down a session just to create a mock, but maybe that is unavoidable.


Was there any thought to making an even stricter mode? When I was working on a reproduction to demonstrate using MockitoSession I kept failing because the mock was not as strict as I was expecting. For example, I'd want the following to fail.

public final class MockTest
{
   @Test
   public void doTest() {
      final MockitoSession session = Mockito.mockitoSession().strictness(Strictness.STRICT_STUBS).startMocking();
      final IWhatever mock = Mockito.mock(IWhatever.class);
      Assert.assertNull(mock.getResult("foo"));
   }
}

interface IWhatever {
   public String getResult(String foo);
}

and

public final class MockTest
{
   @Test
   public void doTest() {
      final MockitoSession session = Mockito.mockitoSession().strictness(Strictness.STRICT_STUBS).startMocking();
      final IWhatever mock = Mockito.mock(IWhatever.class);
      Assert.assertNull(mock.getResult());
   }
}

interface IWhatever {
   public String getResult();
}

In both cases the test will pass when I would want the strict mocks to fail because I didn't configure the methods invoked.

@cosenmarco
Copy link

cosenmarco commented Mar 12, 2019

One case where this is clearly controversial is in the case where we extend some base class which is setting up some default basic stubbing for some (but not all) of the child classes. I know using inheritance in tests is an anti-pattern but the team has historically doing this and a refactoring is out of scope. Is there a way to address this without having to use org.mockito.runners.MockitoJUnitRunner.Silent? I'd like still to keep the benefits of strictness but just be lenient when some stubs are used by some of the child classes

@Druckles
Copy link

Druckles commented Jul 25, 2019

While writing an integration test I came across a specific example where this Strictness may actually be useful. In this example, I want to explicitly mock a call to a service and ensure it is definitely called. I have, for example, the following method:

public void unzipRemotely(String location, String filename) {
    File original = service.download(location, filename);
    ZipFile zip = new ZipFile(original);
    File tempDirectory = createTempDirectory();
    zip.unzip(tempDirectory);
    service.uploadDirectory(location, removeExtension(filename), tempDirectory, true);
}

The two things I need to test here are:

  • Were the files unzipped?
  • Was uploadDirectory called?

I don't extrinsically care whether download was called; it's implied by the other two facts.

To test this, I need something like the following:

@Test
@DisplayName("")
void success() {
    when(service.download("somewhere", "file.zip")).thenReturn(zippedFile);
    when(service.uploadDirectory(eq("somewhere"), eq("file"), any(File.class), eq(true)))
        .thenAnswer(invocationOnMock -> {
            File sourceDirectory = invocationOnMock.getArgument(2);
            assertThatAllFilesAreThere(sourceDirectory);
        });

    unzipper.unzipRemotely("somewhere", "file.zip");
}

The problem here is that there is no guarantee that uploadDirectory was ever called. There are three existing solutions to this:

  • Assert that some flag is set
  • Rely on Strict mode
  • verify

Assert some flag is set

AtomicBoolean wasCalled = new AtomicBoolean(false);
when(service. ...)
    .thenAnswer(invocationOnMock -> {
        // ...
        wasCalled.set(true);
    });
assertThat(wasCalled, is(true));

Obvious problem with this: it requires more code. It's also not as succinct for describing what is happening.

Rely on Strict mode

This is the current solution. You called when. You should expect it to be called.

Possible issues:

  • Anyone can come along and add LENIENT over the tests to expand them. It's no longer verified.
  • It's not explicit from the name. "When" is pretty close to "if". In fact the Germans don't even distinguish between the two. In English, "when" means both, "when this thing definitely occurs in the future" and also more generally "whenever this happens". If you say, "when trees fall they make a noise", it doesn't mean a tree in that forest over there will fall.

Verify

The most natural of the options, I want to explicitly verify that something happened. But for that, it need the following line:

verify(service).uploadDirectory(eq("somewhere"), eq("file"), any(File.class), eq(true));

Problem: duplicated code. Pretty obvious that that's not ideal.

An alternative solution

What about instead of changing the API (which is what Strict-mode does; it means you have to change all your tests in some way), the API is expanded with an extra command:

expect(service.uploadDirectory(eq("somewhere"), eq("file"), any(File.class), eq(true)))
    .andAnswer(...);
  • Works just like when but must be called at least once
  • Like verify it accepts a VerificationMode like times
  • Explicitly describes what it does
  • when no longer verifies that the stubbed method was called

This would solve all the above issues. Non-mocked method calls could still throw an exception for missing stubbings (which does indicate a problem).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests