New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strictness in Mockito #769

Open
mockitoguy opened this Issue Nov 19, 2016 · 20 comments

Comments

Projects
None yet
9 participants
@mockitoguy
Member

mockitoguy commented Nov 19, 2016

Introduction

This issue explores the addition of "strict" behavior and APIs to Mockito. Traditionally, Mockito is a lenient mocking framework. For background and motivation, check out Szczepan's short article on LinkedIn or in-depth article on Mockito blog.

Why strictness in Mockito:

  • productivity - improved debugging and writing tests. Currently MockitoHint warns about stubbing argument mismatches. Strict stubbing would promote the warning into an exception, making the test fail fast and improve debuggability / productivity.
  • clean code - avoid unnecessary stubbings, test fails when unncessary stubs are present.

Concrete API proposals:

  • "strict stubbing" proposal: #770
  • "strict mocking" proposal: #1097
  • opt-out from strict stubbing at mock / method level: #792

Example exception message indicating subbing problem:

org.mockito.exceptions.misusing.PotentialStubbingProblem:
Strict stubbing argument mismatch. Please check:
- this invocation of 'simpleMethod' method:
    mock.simpleMethod(“Foo");
      -> at … StrictStubbingEndToEndTest.java:101
- has following stubbing(s) with different arguments:
    1. mock.simpleMethod(“foo");
      -> at … StrictStubbingEndToEndTest.java:100
Typically, stubbing argument mismatch indicates user mistake when writing tests.
Mockito fails early so that you can debug potential problem easily.

Details

Strictness in Mockito can have 2 forms:

  • Strict stubbing that requires that all declared stubs are actually used
  • Strict mocks, that require all non-void methods to be stubbed before they are called

Strictness behavior can be implemented as:

  • warning on the console
  • exception thrown

Strictness API can be implemented in:

  • JUnit runner / rule
  • TestNG runner
  • explicit method to call by the user

Future direction:

  • strict stubbing on by default, opt-out available
  • strict mocking off by default, opt-in available

Implementation as of Mockito 2.2.x:

  • JUnitRunner - fails at the end of test class when unused stubbings are present. Opt out available via JUnitRunner.Silent
  • JUnitRule - warn at the end of test method about a) unused stubs b) stubs invoked with different arguments. Opt-out available via rule().silent().

Mockito 2.x proposal:

  1. New "strict stubbing" API for JUnit rule - #770
  2. New Answer.StrictMock that fails immediately when non-void method invoked on without prior stubbing - #649
  3. new verifyStubs() method to use when rule/runner not in used

Mockito 3 proposal:

  • strict stubbing is the default
  • change behavior of verifyNoMoreInvocations()
  • runner and rule behave the same

mockitoguy added a commit that referenced this issue Nov 19, 2016

Refactoring JUnit rule implementation
Needed for work on Mockito strictness, see #769. Decoupled JUnit API from the work Mockito does before and after every test.

No public API exposed, listeners and friends need to be refactored and documented.

Don't merge. This refactoring on its own does not add value.
@imod

This comment has been minimized.

Show comment
Hide comment
@imod

imod Nov 21, 2016

I think from a code maintenance point of view, "behavior: warning on the console" is not really useful - in such a case a CI system would not inform anyone about unused stubs. An exception would be a lot better.

imod commented Nov 21, 2016

I think from a code maintenance point of view, "behavior: warning on the console" is not really useful - in such a case a CI system would not inform anyone about unused stubs. An exception would be a lot better.

mockitoguy added a commit that referenced this issue Dec 10, 2016

Refactoring JUnit rule implementation
Needed for work on Mockito strictness, see #769. Decoupled JUnit API from the work Mockito does before and after every test.

No public API exposed, listeners and friends need to be refactored and documented.

Don't merge. This refactoring on its own does not add value.

mockitoguy added a commit that referenced this issue Dec 11, 2016

Refactoring JUnit rule implementation
Needed for work on Mockito strictness, see #769. Decoupled JUnit API from the work Mockito does before and after every test.

No public API exposed, listeners and friends need to be refactored and documented.

Don't merge. This refactoring on its own does not add value.
@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Dec 13, 2016

Contributor

@szczepiq could you add list items (- [ ] in markdown) to signify what the remaining action points in this issue are? That gives a clearer overview, especially as multiple open issues are related to this single epic.

Contributor

TimvdLippe commented Dec 13, 2016

@szczepiq could you add list items (- [ ] in markdown) to signify what the remaining action points in this issue are? That gives a clearer overview, especially as multiple open issues are related to this single epic.

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Jan 13, 2017

Member

@szczepiq could you add list items (- [ ] in markdown) to signify what the remaining action points in this issue are? That gives a clearer overview, especially as multiple open issues are related to this single epic.

I will do that once this epic turns into an execution plan. Currently it is intended to scope out the details and I don't know if all of them will be executed (and in what form). I annotated one item as merged by using strikethrough. I hope it is clearer now.

Member

mockitoguy commented Jan 13, 2017

@szczepiq could you add list items (- [ ] in markdown) to signify what the remaining action points in this issue are? That gives a clearer overview, especially as multiple open issues are related to this single epic.

I will do that once this epic turns into an execution plan. Currently it is intended to scope out the details and I don't know if all of them will be executed (and in what form). I annotated one item as merged by using strikethrough. I hope it is clearer now.

@micheljung

This comment has been minimized.

Show comment
Hide comment
@micheljung

micheljung Jan 31, 2017

Since your Javadoc asks for feedback, here's my case:

  @Test
  public void testAutoCompleteDoesntCompleteWhenTheresNoWordBeforeCaret() throws Exception {
    when(playerService.getPlayerNames()).thenReturn(FXCollections.observableSet("DummyUser", "Junit"));
    textInputControl.setText("j");
    textInputControl.positionCaret(0);

    simulate(keyEvent(KeyCode.TAB));

    assertThat(textInputControl.getText(), is("j"));
  }

It tests auto-completion of an input field.

getPlayerNames() is stubbed, but never called in working code - because if the caret is at position 0, there should be no autocompletion. If I remove the stubbing it's possible to break the autocompletion without being detected; the test would succeed not because it's properly implemented, but because there are no users being returned.

Update:
After hitting enter I realized that I can improve this using verify(playerService, never()).getPlayerNames().
I decided to not remove the comment though :)

micheljung commented Jan 31, 2017

Since your Javadoc asks for feedback, here's my case:

  @Test
  public void testAutoCompleteDoesntCompleteWhenTheresNoWordBeforeCaret() throws Exception {
    when(playerService.getPlayerNames()).thenReturn(FXCollections.observableSet("DummyUser", "Junit"));
    textInputControl.setText("j");
    textInputControl.positionCaret(0);

    simulate(keyEvent(KeyCode.TAB));

    assertThat(textInputControl.getText(), is("j"));
  }

It tests auto-completion of an input field.

getPlayerNames() is stubbed, but never called in working code - because if the caret is at position 0, there should be no autocompletion. If I remove the stubbing it's possible to break the autocompletion without being detected; the test would succeed not because it's properly implemented, but because there are no users being returned.

Update:
After hitting enter I realized that I can improve this using verify(playerService, never()).getPlayerNames().
I decided to not remove the comment though :)

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Feb 1, 2017

Member

@micheljung, my recommendation for this scenario is exactly using verify + never :) I will assume that your use case is satisfied. Looks like strict stubbing nudged you to improve this test!

Thank you very much for reporting. This is exactly kind of feedback we're looking for when evaluating strictness feature.

Member

mockitoguy commented Feb 1, 2017

@micheljung, my recommendation for this scenario is exactly using verify + never :) I will assume that your use case is satisfied. Looks like strict stubbing nudged you to improve this test!

Thank you very much for reporting. This is exactly kind of feedback we're looking for when evaluating strictness feature.

@skestle

This comment has been minimized.

Show comment
Hide comment
@skestle

skestle Jul 21, 2017

I would like an option to verifyNoMoreInteractions in a smart way:

  • ignoring expected/stub invocations as in STRICT_STUBS mode
  • ignoring unnecessary stubs methods as in non-strict modes.

I'm simulating an external API (Selenium) to ensure I wrap it properly.

	public void expectToFindNoElement(final By by) {
		when(this.webElement.findElement(by)).thenThrow(new NoSuchElementException("Cannot locate an element " + by));
		when(this.webElement.findElements(by)).thenReturn(Collections.emptyList());
	}

It's very easy to accidentally not throw the exception when we're expecting no element; the correct thing selenium consumers generally want to do is verify no elements.

Essentially, I can't be DRY with Strict without breaking it's strict "clean and maintainable" expectations.

BTW, I agree this is not "clean" but it is certainly very much more maintainable (and correct) than what STRICT mode allows.

skestle commented Jul 21, 2017

I would like an option to verifyNoMoreInteractions in a smart way:

  • ignoring expected/stub invocations as in STRICT_STUBS mode
  • ignoring unnecessary stubs methods as in non-strict modes.

I'm simulating an external API (Selenium) to ensure I wrap it properly.

	public void expectToFindNoElement(final By by) {
		when(this.webElement.findElement(by)).thenThrow(new NoSuchElementException("Cannot locate an element " + by));
		when(this.webElement.findElements(by)).thenReturn(Collections.emptyList());
	}

It's very easy to accidentally not throw the exception when we're expecting no element; the correct thing selenium consumers generally want to do is verify no elements.

Essentially, I can't be DRY with Strict without breaking it's strict "clean and maintainable" expectations.

BTW, I agree this is not "clean" but it is certainly very much more maintainable (and correct) than what STRICT mode allows.

@skestle

This comment has been minimized.

Show comment
Hide comment
@skestle

skestle Jul 21, 2017

So in terms of 8.5.47's implementation, I'd like to haveUniversalTestListener.testFinished in LENIENT mode, while DefaultStubbingLookupListener.onStubbingLookup executes invocation.markVerified(), as it does in strict mode.

Probably what I want is DRY without strictness as mockito wants to define it. IMO, DefaultStubbingLookupListener and DRY verifyNoMoreInteractions() has nothing to do with strictness.
Please separate these two concepts.

skestle commented Jul 21, 2017

So in terms of 8.5.47's implementation, I'd like to haveUniversalTestListener.testFinished in LENIENT mode, while DefaultStubbingLookupListener.onStubbingLookup executes invocation.markVerified(), as it does in strict mode.

Probably what I want is DRY without strictness as mockito wants to define it. IMO, DefaultStubbingLookupListener and DRY verifyNoMoreInteractions() has nothing to do with strictness.
Please separate these two concepts.

@skestle

This comment has been minimized.

Show comment
Hide comment
@skestle

skestle Jul 21, 2017

Also, having tried for half an hour trying to grok StubbingLookupListeners, CreationSettings MockitoTestListeners etc, it seems there's no way to insert invocation.markVerified() without creating my own Test Runner.

If there's some sort of workaround where I can make this auto-verification automatic, please respond. Should there be another ticket to add auto-verification of stubbed invocations (which is an awesome feature BTW) that's separate from this ticket (as it should be)?

skestle commented Jul 21, 2017

Also, having tried for half an hour trying to grok StubbingLookupListeners, CreationSettings MockitoTestListeners etc, it seems there's no way to insert invocation.markVerified() without creating my own Test Runner.

If there's some sort of workaround where I can make this auto-verification automatic, please respond. Should there be another ticket to add auto-verification of stubbed invocations (which is an awesome feature BTW) that's separate from this ticket (as it should be)?

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Jul 21, 2017

Member

Thank you for reporting and debugging Mockito strictness! Let me give you some feedback about the use case first:

I would like an option to verifyNoMoreInteractions in a smart way:
ignoring expected/stub invocations as in STRICT_STUBS mode
ignoring unnecessary stubs methods as in non-strict modes.

It seems that you are suggesting a different Strictness level, something between "lenient" and "strict stubs":

  • lenient - allow unused stubbings, veryNoMoreInteractions catches all
  • "kinda strict stubs" - allow unused stubbings, verifyNoMoreInteractions ignores used stubbings
  • strict stubs - disallow unused stubbings, verifyNoMoreInteractions ignores used stubbings

Have you tried this:

verifyNoMoreInteractions(ignoreStubs(mock1, mock2));

Before we jump to the impl details of TestListener (which is an internal type, not intended for regular use, but absolutely open to experimentation and creative usage), let's nail down the use case first.

Thank you for reporting! This is a very interesting use case.

Member

mockitoguy commented Jul 21, 2017

Thank you for reporting and debugging Mockito strictness! Let me give you some feedback about the use case first:

I would like an option to verifyNoMoreInteractions in a smart way:
ignoring expected/stub invocations as in STRICT_STUBS mode
ignoring unnecessary stubs methods as in non-strict modes.

It seems that you are suggesting a different Strictness level, something between "lenient" and "strict stubs":

  • lenient - allow unused stubbings, veryNoMoreInteractions catches all
  • "kinda strict stubs" - allow unused stubbings, verifyNoMoreInteractions ignores used stubbings
  • strict stubs - disallow unused stubbings, verifyNoMoreInteractions ignores used stubbings

Have you tried this:

verifyNoMoreInteractions(ignoreStubs(mock1, mock2));

Before we jump to the impl details of TestListener (which is an internal type, not intended for regular use, but absolutely open to experimentation and creative usage), let's nail down the use case first.

Thank you for reporting! This is a very interesting use case.

@karollewandowski

This comment has been minimized.

Show comment
Hide comment
@karollewandowski

karollewandowski Aug 22, 2017

I like strict stubbing very much.
I've seen opinion that it allows to remove a lot of code, but I think that it rather forces to write a little more. I used to put common stubbing code to setup method even if it wasn't used by all tests. Now I have to move it to tests actually using it. I think that it's nothing bad and makes my tests clearer. When stubbing code is in test method, then there is no need to scroll source file to top to check what is stubbed and what is not. Everything is in one place and test context is explicit and close to it's usage. More code doesn't mean duplications - common stubbing can be extracted to method with descriptive name.
@szczepiq, on Twitter you mentioned that you are wondering API like when(method, optional()).then(...). I think that it's not necessary, but sometimes can be helpful.
It would allow to create optional stubbing in setup method what would break clarity provided by strict stubbing again. On the other hand, sometimes we don't care about some dependency or there are tens of tests using some stubbing and one not, so optional() would be nice to handle such cases (and it clearly says it's optional).

karollewandowski commented Aug 22, 2017

I like strict stubbing very much.
I've seen opinion that it allows to remove a lot of code, but I think that it rather forces to write a little more. I used to put common stubbing code to setup method even if it wasn't used by all tests. Now I have to move it to tests actually using it. I think that it's nothing bad and makes my tests clearer. When stubbing code is in test method, then there is no need to scroll source file to top to check what is stubbed and what is not. Everything is in one place and test context is explicit and close to it's usage. More code doesn't mean duplications - common stubbing can be extracted to method with descriptive name.
@szczepiq, on Twitter you mentioned that you are wondering API like when(method, optional()).then(...). I think that it's not necessary, but sometimes can be helpful.
It would allow to create optional stubbing in setup method what would break clarity provided by strict stubbing again. On the other hand, sometimes we don't care about some dependency or there are tens of tests using some stubbing and one not, so optional() would be nice to handle such cases (and it clearly says it's optional).

@adamvoss

This comment has been minimized.

Show comment
Hide comment
@adamvoss

adamvoss Jan 30, 2018

Can this be exposed by MockSettings or otherwise be made available without requiring the Mock instance be a class member.

The following "works" but uses a whole lot more internal stuff than I like:

import org.mockito.MockSettings;
import org.mockito.Mockito;
import org.mockito.internal.creation.settings.CreationSettings;
import org.mockito.internal.listeners.StubbingLookupListener;
import org.mockito.mock.MockCreationSettings;
import org.mockito.quality.Strictness;

import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;

import static org.mockito.Mockito.withSettings;

public final class StrictMockito
{
   private StrictMockito()
   {
   }

   public static <T> T strictMock(Class<T> classToMock) {
      final MockSettings mockSettings = withSettings();
      createDefaultStubbingLookupListener(Strictness.STRICT_STUBS);
      final T mock = Mockito.mock(classToMock, mockSettings);
      final MockCreationSettings<?> mockCreationSettings = Mockito.mockingDetails(mock).getMockCreationSettings();
      ((CreationSettings<?>)mockCreationSettings).getStubbingLookupListeners().add( createDefaultStubbingLookupListener(Strictness.STRICT_STUBS));
      return mock;
   }

   private static StubbingLookupListener createDefaultStubbingLookupListener(Strictness strictness)
   {
      try
      {
         final Class<?> clazz = Class.forName("org.mockito.internal.junit.DefaultStubbingLookupListener");
         final Constructor<?> constructor = clazz.getDeclaredConstructor(Strictness.class);
         constructor.setAccessible(true);
         return (StubbingLookupListener) constructor.newInstance(strictness);
      }
      catch (ClassNotFoundException | NoSuchMethodException | IllegalAccessException | InstantiationException | InvocationTargetException e)
      {
         throw new RuntimeException(e);
      }
   }
}

adamvoss commented Jan 30, 2018

Can this be exposed by MockSettings or otherwise be made available without requiring the Mock instance be a class member.

The following "works" but uses a whole lot more internal stuff than I like:

import org.mockito.MockSettings;
import org.mockito.Mockito;
import org.mockito.internal.creation.settings.CreationSettings;
import org.mockito.internal.listeners.StubbingLookupListener;
import org.mockito.mock.MockCreationSettings;
import org.mockito.quality.Strictness;

import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;

import static org.mockito.Mockito.withSettings;

public final class StrictMockito
{
   private StrictMockito()
   {
   }

   public static <T> T strictMock(Class<T> classToMock) {
      final MockSettings mockSettings = withSettings();
      createDefaultStubbingLookupListener(Strictness.STRICT_STUBS);
      final T mock = Mockito.mock(classToMock, mockSettings);
      final MockCreationSettings<?> mockCreationSettings = Mockito.mockingDetails(mock).getMockCreationSettings();
      ((CreationSettings<?>)mockCreationSettings).getStubbingLookupListeners().add( createDefaultStubbingLookupListener(Strictness.STRICT_STUBS));
      return mock;
   }

   private static StubbingLookupListener createDefaultStubbingLookupListener(Strictness strictness)
   {
      try
      {
         final Class<?> clazz = Class.forName("org.mockito.internal.junit.DefaultStubbingLookupListener");
         final Constructor<?> constructor = clazz.getDeclaredConstructor(Strictness.class);
         constructor.setAccessible(true);
         return (StubbingLookupListener) constructor.newInstance(strictness);
      }
      catch (ClassNotFoundException | NoSuchMethodException | IllegalAccessException | InstantiationException | InvocationTargetException e)
      {
         throw new RuntimeException(e);
      }
   }
}
@adamvoss

This comment has been minimized.

Show comment
Hide comment
@adamvoss

adamvoss Jan 30, 2018

If possible, the failure message around passing null and collections could use improvement. As is, we can get errors like the following:

Strict stubbing argument mismatch. Please check:
 - this invocation of 'getValue' method:
    sut.getValue("", null, []);
    -> at ExampleTest.strictExample(ExampleTest.java:64)
 - has following stubbing(s) with different arguments:
    1. sut.getValue("", null, null);
      -> at ExampleTest.strictExample(ExampleTest.java:62)

It is not clear from the failure message that only the middle argument is the source of the problem. One is likely to expect the final argument is the problem because it is the entry that is different.

Code to reproduce the above:

   public interface Sut
   {
      Object getValue(String foo, Object bar, Set<String> baz);
   }

   @Mock
   Sut sut;

   @Test
   public void strictExample() {
      Mockito.when(sut.getValue(ArgumentMatchers.anyString(), ArgumentMatchers.any(Object.class),
            ArgumentMatchers.eq(Collections.emptySet()))).thenReturn(null);

      sut.getValue("", null, Collections.emptySet());
   }

adamvoss commented Jan 30, 2018

If possible, the failure message around passing null and collections could use improvement. As is, we can get errors like the following:

Strict stubbing argument mismatch. Please check:
 - this invocation of 'getValue' method:
    sut.getValue("", null, []);
    -> at ExampleTest.strictExample(ExampleTest.java:64)
 - has following stubbing(s) with different arguments:
    1. sut.getValue("", null, null);
      -> at ExampleTest.strictExample(ExampleTest.java:62)

It is not clear from the failure message that only the middle argument is the source of the problem. One is likely to expect the final argument is the problem because it is the entry that is different.

Code to reproduce the above:

   public interface Sut
   {
      Object getValue(String foo, Object bar, Set<String> baz);
   }

   @Mock
   Sut sut;

   @Test
   public void strictExample() {
      Mockito.when(sut.getValue(ArgumentMatchers.anyString(), ArgumentMatchers.any(Object.class),
            ArgumentMatchers.eq(Collections.emptySet()))).thenReturn(null);

      sut.getValue("", null, Collections.emptySet());
   }
@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Jan 31, 2018

Member

Hey @adamvoss! Thanks for checking in!

Can this be exposed by MockSettings or otherwise be made available without requiring the Mock instance be a class member

Mocks don't have to be class members to leverage strictness. You can use MockitoSession object to start mocking before you create mocks. Would that solve your problem?

Member

mockitoguy commented Jan 31, 2018

Hey @adamvoss! Thanks for checking in!

Can this be exposed by MockSettings or otherwise be made available without requiring the Mock instance be a class member

Mocks don't have to be class members to leverage strictness. You can use MockitoSession object to start mocking before you create mocks. Would that solve your problem?

@adamvoss

This comment has been minimized.

Show comment
Hide comment
@adamvoss

adamvoss Feb 16, 2018

@mockitoguy Sorry, I missed the notification on this one.

Here is what that looks like:

public final class MockTest
{
   @Test
   public void doTest() {
      final MockitoSession session = Mockito.mockitoSession().strictness(Strictness.STRICT_STUBS).startMocking();
      final IWhatever mock = Mockito.mock(IWhatever.class);
      when(mock.getResult("bar")).thenReturn("yes");
      Assert.assertNotNull(mock.getResult("foo"));
      
      // Should be in tear down (after) method
      session.finishMocking();
   }
}


interface IWhatever {
   public String getResult(String foo);
}

I would say that is more flexible, but not exactly what I was looking for. One of the motivations being able to specify it on a per-mock setting is to be able to introduce them gradually. That way I can use strict when I want to (including existing tests) without changing any of the existing mocks.

With you wanting to make this eventually the default mode, then I can see the argument for not allowing granular configuration. (Though once on v3, people may want to gradually use some lenient mocks). That aside, it is an annoyance to need to setup and tear down a session just to create a mock, but maybe that is unavoidable.


Was there any thought to making an even stricter mode? When I was working on a reproduction to demonstrate using MockitoSession I kept failing because the mock was not as strict as I was expecting. For example, I'd want the following to fail.

public final class MockTest
{
   @Test
   public void doTest() {
      final MockitoSession session = Mockito.mockitoSession().strictness(Strictness.STRICT_STUBS).startMocking();
      final IWhatever mock = Mockito.mock(IWhatever.class);
      Assert.assertNull(mock.getResult("foo"));
   }
}

interface IWhatever {
   public String getResult(String foo);
}

and

public final class MockTest
{
   @Test
   public void doTest() {
      final MockitoSession session = Mockito.mockitoSession().strictness(Strictness.STRICT_STUBS).startMocking();
      final IWhatever mock = Mockito.mock(IWhatever.class);
      Assert.assertNull(mock.getResult());
   }
}

interface IWhatever {
   public String getResult();
}

In both cases the test will pass when I would want the strict mocks to fail because I didn't configure the methods invoked.

adamvoss commented Feb 16, 2018

@mockitoguy Sorry, I missed the notification on this one.

Here is what that looks like:

public final class MockTest
{
   @Test
   public void doTest() {
      final MockitoSession session = Mockito.mockitoSession().strictness(Strictness.STRICT_STUBS).startMocking();
      final IWhatever mock = Mockito.mock(IWhatever.class);
      when(mock.getResult("bar")).thenReturn("yes");
      Assert.assertNotNull(mock.getResult("foo"));
      
      // Should be in tear down (after) method
      session.finishMocking();
   }
}


interface IWhatever {
   public String getResult(String foo);
}

I would say that is more flexible, but not exactly what I was looking for. One of the motivations being able to specify it on a per-mock setting is to be able to introduce them gradually. That way I can use strict when I want to (including existing tests) without changing any of the existing mocks.

With you wanting to make this eventually the default mode, then I can see the argument for not allowing granular configuration. (Though once on v3, people may want to gradually use some lenient mocks). That aside, it is an annoyance to need to setup and tear down a session just to create a mock, but maybe that is unavoidable.


Was there any thought to making an even stricter mode? When I was working on a reproduction to demonstrate using MockitoSession I kept failing because the mock was not as strict as I was expecting. For example, I'd want the following to fail.

public final class MockTest
{
   @Test
   public void doTest() {
      final MockitoSession session = Mockito.mockitoSession().strictness(Strictness.STRICT_STUBS).startMocking();
      final IWhatever mock = Mockito.mock(IWhatever.class);
      Assert.assertNull(mock.getResult("foo"));
   }
}

interface IWhatever {
   public String getResult(String foo);
}

and

public final class MockTest
{
   @Test
   public void doTest() {
      final MockitoSession session = Mockito.mockitoSession().strictness(Strictness.STRICT_STUBS).startMocking();
      final IWhatever mock = Mockito.mock(IWhatever.class);
      Assert.assertNull(mock.getResult());
   }
}

interface IWhatever {
   public String getResult();
}

In both cases the test will pass when I would want the strict mocks to fail because I didn't configure the methods invoked.

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Feb 19, 2018

Member

Was there any thought to making an even stricter mode?

Yes. Strict mocks (different beasts than strict stubs): #1097. You may want to check out the overview of strictness: #769

Member

mockitoguy commented Feb 19, 2018

Was there any thought to making an even stricter mode?

Yes. Strict mocks (different beasts than strict stubs): #1097. You may want to check out the overview of strictness: #769

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Feb 19, 2018

Member

One of the motivations being able to specify it on a per-mock setting is to be able to introduce them gradually.

We started working on per-mock / per-stubbing strictness in: #792, check it out!

Thank you for insightful feedback!

Member

mockitoguy commented Feb 19, 2018

One of the motivations being able to specify it on a per-mock setting is to be able to introduce them gradually.

We started working on per-mock / per-stubbing strictness in: #792, check it out!

Thank you for insightful feedback!

@ahasbini

This comment has been minimized.

Show comment
Hide comment
@ahasbini

ahasbini Mar 25, 2018

Hello,

I'm having a hard time figuring this out and would hope if someone could help me as I am using Strict Stubbing with slightly complicated case.

I'm currently writing instrumentation test cases for an Android application which uses mockito to "sort of" unit test certain components such fragments. On top of that I have obfuscation enabled as I am trying to automate testing of the entire code when it is obfuscated to catch obfuscation problems early.

My problem is a certain interface has two methods that are of different names and different parameters. I've implemented stubbing for one of the methods properly with Strict Stub rule enabled and obfuscation first disabled. The test got executed successfully. Once I turned on the obfuscation, these two methods got renamed to the same letter 'a', hence they are now overloaded. I re-ran the test and got the PotentionalStubbingProblem since the other unstubbed method with different parameter signature got called before the stubbed method.

Here's an example:

package com.example;

public interface MyInterface {
    boolean method1();
    String method2(String i, String j);
}

Test method:

// Assuming that mockito annotations where initialized properly and Strict Stub rule applied

@Mock
private MyInterface mockedInterface;

@Test
public void exampleTest() {
    Mockito.when(mockedInterface.method2(Mockit.<String>any(), Mockit.<String>any())).thenReturn("fake_string");
    // Pass the mocked interface to a class that depends on it and interact with that class which
    // will call the methods within the mocked object.
    // At a certain point, when method1 was called, Mockito throws a PotentionalStubbingProblem
    // exception at the same line of stubbing for method2, with the stacktrace containing both
    // methods with the same overloaded name and their different parameter signatures.
}

Hope this is giving an idea of what I'm facing. I'm currently using mockito-android:2.16.0 in my application. Although this might not be an obfuscation problem, could this be an overloading problem?

Thanks!

ahasbini commented Mar 25, 2018

Hello,

I'm having a hard time figuring this out and would hope if someone could help me as I am using Strict Stubbing with slightly complicated case.

I'm currently writing instrumentation test cases for an Android application which uses mockito to "sort of" unit test certain components such fragments. On top of that I have obfuscation enabled as I am trying to automate testing of the entire code when it is obfuscated to catch obfuscation problems early.

My problem is a certain interface has two methods that are of different names and different parameters. I've implemented stubbing for one of the methods properly with Strict Stub rule enabled and obfuscation first disabled. The test got executed successfully. Once I turned on the obfuscation, these two methods got renamed to the same letter 'a', hence they are now overloaded. I re-ran the test and got the PotentionalStubbingProblem since the other unstubbed method with different parameter signature got called before the stubbed method.

Here's an example:

package com.example;

public interface MyInterface {
    boolean method1();
    String method2(String i, String j);
}

Test method:

// Assuming that mockito annotations where initialized properly and Strict Stub rule applied

@Mock
private MyInterface mockedInterface;

@Test
public void exampleTest() {
    Mockito.when(mockedInterface.method2(Mockit.<String>any(), Mockit.<String>any())).thenReturn("fake_string");
    // Pass the mocked interface to a class that depends on it and interact with that class which
    // will call the methods within the mocked object.
    // At a certain point, when method1 was called, Mockito throws a PotentionalStubbingProblem
    // exception at the same line of stubbing for method2, with the stacktrace containing both
    // methods with the same overloaded name and their different parameter signatures.
}

Hope this is giving an idea of what I'm facing. I'm currently using mockito-android:2.16.0 in my application. Although this might not be an obfuscation problem, could this be an overloading problem?

Thanks!

@ahasbini

This comment has been minimized.

Show comment
Hide comment
@ahasbini

ahasbini Mar 27, 2018

Hi @mockitoguy, sorry to bother you if any. Was just hoping if you could provide me some feedback whether I'm on the right path with this issue or not.

I've done another test with obfuscation completely disabled, but a class has two methods with the same name but different parameters signature. I've applied the Strict Stub rule and got the same error below:

org.mockito.exceptions.misusing.PotentialStubbingProblem: 
Strict stubbing argument mismatch. Please check:
 - this invocation of 'myMethod' method:
    myInterface.myMethod();
    -> at test.ahasbini.com.test.Example.start(Example.java:20)
 - has following stubbing(s) with different arguments:
    1. myInterface.myMethod("", "");
      -> at test.ahasbini.com.test.MockitoOverloadInstrumentationTest.testOverload(MockitoOverloadInstrumentationTest.java:33)
Typically, stubbing argument mismatch indicates user mistake when writing tests.
Mockito fails early so that you can debug potential problem easily.
However, there are legit scenarios when this exception generates false negative signal:
  - stubbing the same method multiple times using 'given().will()' or 'when().then()' API
    Please use 'will().given()' or 'doReturn().when()' API for stubbing.
  - stubbed method is intentionally invoked with different arguments by code under test
    Please use default or 'silent' JUnit Rule (equivalent of Strictness.LENIENT).
For more information see javadoc for PotentialStubbingProblem class.
    at test.ahasbini.com.test.Example.start(Example.java:20)
    at test.ahasbini.com.test.MockitoOverloadInstrumentationTest.testOverload(MockitoOverloadInstrumentationTest.java:37)
    at java.lang.reflect.Method.invoke(Native Method)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at android.support.test.internal.runner.junit4.statement.RunBefores.evaluate(RunBefores.java:80)
    at org.mockito.internal.junit.JUnitRule$1.evaluateSafely(JUnitRule.java:52)
    at org.mockito.internal.junit.JUnitRule$1.evaluate(JUnitRule.java:43)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at test.ahasbini.com.test.PermissionRule$1.evaluate(PermissionRule.java:156)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:27)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
    at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:58)
    at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:375)
    at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1932)

Here's my test:

public class MockitoOverloadInstrumentationTest extends BaseTest {

    @Mock
    private MyInterface myInterface;

    @Rule
    public MockitoRule mockitoRule = MockitoJUnit.rule().strictness(Strictness.STRICT_STUBS);

    @Before
    public void initMocks() {
        MockitoAnnotations.initMocks(this);
    }

    @Test
    public void testOverload() {
        // Set Up
        Mockito.when(myInterface                                // <-- Exception happening here
                .myMethod(Mockito.anyString(), Mockito.anyString()))
                .thenReturn("Mocked Hello World!");
        Example example = new Example(myInterface);

        // Test
        example.start();
        Mockito.verify(myInterface, Mockito.timeout(10000L))
                .myMethod(Mockito.anyString(), Mockito.anyString());
    }
}

and the mocked interface below:

public interface MyInterface {

    boolean myMethod();

    String myMethod(String i, String j);
}

and finally the example class:

public class Example {

    private static final String TAG = Example.class.getSimpleName();

    private final MyInterface myInterface;

    public Example(MyInterface myInterface) {
        this.myInterface = myInterface;
    }

    public void start() {
        if (!myInterface.myMethod()) {     // <-- Exception pointing to here as well.
            Log.i(TAG, "start: " + myInterface.myMethod("hello", "world"));
        }
    }

}

ahasbini commented Mar 27, 2018

Hi @mockitoguy, sorry to bother you if any. Was just hoping if you could provide me some feedback whether I'm on the right path with this issue or not.

I've done another test with obfuscation completely disabled, but a class has two methods with the same name but different parameters signature. I've applied the Strict Stub rule and got the same error below:

org.mockito.exceptions.misusing.PotentialStubbingProblem: 
Strict stubbing argument mismatch. Please check:
 - this invocation of 'myMethod' method:
    myInterface.myMethod();
    -> at test.ahasbini.com.test.Example.start(Example.java:20)
 - has following stubbing(s) with different arguments:
    1. myInterface.myMethod("", "");
      -> at test.ahasbini.com.test.MockitoOverloadInstrumentationTest.testOverload(MockitoOverloadInstrumentationTest.java:33)
Typically, stubbing argument mismatch indicates user mistake when writing tests.
Mockito fails early so that you can debug potential problem easily.
However, there are legit scenarios when this exception generates false negative signal:
  - stubbing the same method multiple times using 'given().will()' or 'when().then()' API
    Please use 'will().given()' or 'doReturn().when()' API for stubbing.
  - stubbed method is intentionally invoked with different arguments by code under test
    Please use default or 'silent' JUnit Rule (equivalent of Strictness.LENIENT).
For more information see javadoc for PotentialStubbingProblem class.
    at test.ahasbini.com.test.Example.start(Example.java:20)
    at test.ahasbini.com.test.MockitoOverloadInstrumentationTest.testOverload(MockitoOverloadInstrumentationTest.java:37)
    at java.lang.reflect.Method.invoke(Native Method)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at android.support.test.internal.runner.junit4.statement.RunBefores.evaluate(RunBefores.java:80)
    at org.mockito.internal.junit.JUnitRule$1.evaluateSafely(JUnitRule.java:52)
    at org.mockito.internal.junit.JUnitRule$1.evaluate(JUnitRule.java:43)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at test.ahasbini.com.test.PermissionRule$1.evaluate(PermissionRule.java:156)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:27)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
    at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:58)
    at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:375)
    at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1932)

Here's my test:

public class MockitoOverloadInstrumentationTest extends BaseTest {

    @Mock
    private MyInterface myInterface;

    @Rule
    public MockitoRule mockitoRule = MockitoJUnit.rule().strictness(Strictness.STRICT_STUBS);

    @Before
    public void initMocks() {
        MockitoAnnotations.initMocks(this);
    }

    @Test
    public void testOverload() {
        // Set Up
        Mockito.when(myInterface                                // <-- Exception happening here
                .myMethod(Mockito.anyString(), Mockito.anyString()))
                .thenReturn("Mocked Hello World!");
        Example example = new Example(myInterface);

        // Test
        example.start();
        Mockito.verify(myInterface, Mockito.timeout(10000L))
                .myMethod(Mockito.anyString(), Mockito.anyString());
    }
}

and the mocked interface below:

public interface MyInterface {

    boolean myMethod();

    String myMethod(String i, String j);
}

and finally the example class:

public class Example {

    private static final String TAG = Example.class.getSimpleName();

    private final MyInterface myInterface;

    public Example(MyInterface myInterface) {
        this.myInterface = myInterface;
    }

    public void start() {
        if (!myInterface.myMethod()) {     // <-- Exception pointing to here as well.
            Log.i(TAG, "start: " + myInterface.myMethod("hello", "world"));
        }
    }

}
@ahasbini

This comment has been minimized.

Show comment
Hide comment
@ahasbini

ahasbini Mar 27, 2018

Sorry, might have send the comment twice by accident.

ahasbini commented Mar 27, 2018

Sorry, might have send the comment twice by accident.

@markusjevringgoeuro

This comment has been minimized.

Show comment
Hide comment
@markusjevringgoeuro

markusjevringgoeuro Sep 18, 2018

In the javadoc for PotentialStubbingProblem, you asked for feedback about this strictness, and here it i:. I think it's conceptually a good thing, however the issue with multiple parameters to stubs and the when().then() pattern (which is infinitely more readable than the doReturn().when()) is really making my life unnecessarily hard. I want the strictness, but I'd also like when().then() to behave in a sane manner without having to tell mockito to be lenient for a whole test class. I get why this might be hard to implement, but I'm hoping that the use of eq() or some other "multi-argument marker" might fix when().then().
In addition, the "Unnecessary stubbings detected" code is not great at what it does. There are many times where the stubbing is not, in fact, unnecessary, so then I have to make mockito lenient to work around it.

markusjevringgoeuro commented Sep 18, 2018

In the javadoc for PotentialStubbingProblem, you asked for feedback about this strictness, and here it i:. I think it's conceptually a good thing, however the issue with multiple parameters to stubs and the when().then() pattern (which is infinitely more readable than the doReturn().when()) is really making my life unnecessarily hard. I want the strictness, but I'd also like when().then() to behave in a sane manner without having to tell mockito to be lenient for a whole test class. I get why this might be hard to implement, but I'm hoping that the use of eq() or some other "multi-argument marker" might fix when().then().
In addition, the "Unnecessary stubbings detected" code is not great at what it does. There are many times where the stubbing is not, in fact, unnecessary, so then I have to make mockito lenient to work around it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment