Add annotation to mark a type as DoNotMock #1833
Conversation
Mocking types that users not own [1] or are severely complicating test logic [2] leads to brittle or wrong tests. In particular, the StackOverflow answer is wrong, as the contract of java.util.Map is violated. When a new key is added to the Map, the stubbed return would be wrong. In Google we have used the DoNotMock annotation via ErrorProne [3] to annotate these types, as well as an internal list of types that can't be mocked (this includes several java.util types). We are using a custom Mockmaker to enforce this on run-time. Based on our successful experience with DoNotMock (we have seen a large reduction in bad/broken tests for types involved), we are proposing to open source this into Mockito itself. The DoNotMock annotation can be added to any type, e.g. classes and interfaces. If, in the type hierarchy of the class-to-be-mocked, there is a type that is annotated with DoNotMock, Mockito will throw a DoNotMockException. This would help preventing issues such as #1827 and #1734 which is in-line with the guidance on our wiki [1]. A follow-up change would allow us to define external types (like the java.util types) that can't be mocked. (We can't add the annotation to the types, as they live in the JDK instead.) [1]: https://github.com/mockito/mockito/wiki/How-to-write-good-tests#dont-mock-a-type-you-dont-own [2]: https://stackoverflow.com/a/15820143 [3]: https://errorprone.info/api/latest/com/google/errorprone/annotations/DoNotMock.html
Codecov Report
@@ Coverage Diff @@
## release/3.x #1833 +/- ##
================================================
+ Coverage 86.71% 87.1% +0.38%
- Complexity 2492 2610 +118
================================================
Files 311 316 +5
Lines 6551 6793 +242
Branches 822 848 +26
================================================
+ Hits 5681 5917 +236
- Misses 674 675 +1
- Partials 196 201 +5
Continue to review full report at Codecov.
|
|
Great submission, thank you!
Fantastic idea. THANK you!
Wow! Interesting use of this API. Can you enumerate other interesting use cases Google has extended Mockito behavior via our public API? I'd love us to offer first class support for such features.
+1 |
|
Request: can we add a new method to override this just in case? Here's a use case: let's say someone incorrectly sets DoNotMock annotation on a type and ships the code in a library. The consumers of the library don't have a good way to remove DoNotMock annotation if they don't own the library code. Thoughts? Example API: mock(Foo.class, withSettings().overrideDoNotMock())We can this API later, no need to include this in this PR. |
| mockTypeToCheck = mockTypeToCheck.getSuperclass(); | ||
| } | ||
|
|
||
| if (unmockableTypes.stream().anyMatch(clazz -> clazz.isAnnotationPresent(DoNotMock.class))) { |
There was a problem hiding this comment.
Do we want to honor this specific annotation (Mockito API) or any annotation with name "DoNotMock".
The latter seems more useful. For example: a library owner can mark types with any "DoNotMock" annotation without adding compile dependency on mockito-core.
There was a problem hiding this comment.
I would like to keep this for now and open it up later to match on name. If we would implement that now, I would break all users of ErrorProne that rely on @DoNotMock but have grandfathered mock violations.
If we can gather feedback first, we can see adoption patterns and change it to match on the name later. WDYT?
There was a problem hiding this comment.
How about a happy medium so that we don't need to revisit this in the future (and cause breaking changes :) - we can match for "org.mockito.DoNotMock" by String. This should be backwards compatible for G. and other companies / projects. Thoughts?
| import org.mockito.MockitoAnnotations; | ||
| import org.mockito.exceptions.misusing.DoNotMockException; | ||
|
|
||
| public class DoNotMockTest { |
Yes I am auditing/discovering usages of Mockito API and reviewing them whether they are worth keeping (and open sourcing) or should be changed. This is one of many 😄
Yes we already have the concept of "grandfathered" mock violations. E.g. all existing mocks of a type that you want to add |
| creationSettings.getExtraInterfaces()); | ||
|
|
||
| Class<?> mockTypeToCheck = typeToMock; | ||
| while (mockTypeToCheck != null && mockTypeToCheck != Object.class) { |
There was a problem hiding this comment.
I played with the code and noticed that "mockTypeToCheck != null" part of this statement is either: a) not needed or b) not covered with tests.
Do we need this part?
There was a problem hiding this comment.
It is covered by mocking an interface that directly has the DoNotMock annotation. I added a test to cover this case.
|
Really interesting feature, also with regards to returning mocks from methods, I think. But very much +1 on the override functionality. There is always a "but" that might need consideration. |
This introduces the DoNotMockEnforcer interface which users can override to implement their special handling of types annotated with DoNotMock.
|
It is now possible to provide a custom Given the positive response, I will now spend some time integrating the changes in Google and see if it all builds. Given that is all positive and you have no more review comments, I will merge this PR. |
This name is clearer for implementers.
Curious, are you going to use this new plugin at G? If we don't have a user, I suggest we avoid complexity (it can be added later if needed). Thank you for implementing name matching for the annotation!!! |
Yes we will immediately use it. We have some additional classes that we are annotating as well as our "grandfathering violations" system. Note that there is no other nice way of overriding the method as requested in #1833 (comment) (a method call per mock creation would mean that we would need to change every existing mock in our codebase to include it, which is very cumbersome)
Glad you like it 😄 |
| * @param type The type to check | ||
| * @return Optional message if this type can not be mocked, or an empty optional if type can be mocked | ||
| */ | ||
| Optional<String> checkTypeForDoNotMockViolation(Class<?> type); |
There was a problem hiding this comment.
So you prefer the enforcer to return the message instead of just throwing?
Works for me. Just checking.
There was a problem hiding this comment.
I felt like this was the nicer way of handling state, since we have to stop type traversal. I will mark it as Incubating and might change later, depending on how well it integrates with our internal system.
| import java.lang.annotation.Target; | ||
|
|
||
| /** | ||
| * Test to make sure that we are matching on name rather than only {@link org.mockito.DoNotMock} |
| public void thrown_exception_includes_special_non_mockable_reason() { | ||
| assertThatThrownBy(() -> { | ||
| NotMockableWithReason notMockable = mock(NotMockableWithReason.class); | ||
| }).isInstanceOf(DoNotMockException.class).hasMessageContaining("Special reason"); |
| annotation -> annotation.annotationType().getName().endsWith("org.mockito.DoNotMock")) | ||
| .findFirst() | ||
| .map(annotation -> { | ||
| String exceptionMessage = type + " is annotated with @DoNoMock and can't be mocked."; |
There was a problem hiding this comment.
do we want to say "is annotated with org.mockito.@DoNotMock"?
There was a problem hiding this comment.
Yes that's better :)
| @@ -50,6 +56,9 @@ | |||
| @SuppressWarnings("unchecked") | |||
| public class MockitoCore { | |||
|
|
|||
| private static final DoNotMockEnforcer DO_NOT_MOCK_ENFORCER = Plugins.getDoNotMockEnforcer(); | |||
| private static final Set<Class<?>> SAFE_DONOTMOCK_ENFORCED_CLASSES = Collections.synchronizedSet(new HashSet<>()); | |||
| checkDoNotMockAnnotationForType(type.getSuperclass()); | ||
| Arrays.stream(type.getInterfaces()).forEach(MockitoCore::checkDoNotMockAnnotationForType); | ||
|
|
||
| SAFE_DONOTMOCK_ENFORCED_CLASSES.add(type); |
There was a problem hiding this comment.
I feel uneasy about this perf tweak. Do we need it? Does it really make things faster? Static state that keeps growing with every mocked type may bite us later.
Did you find this perf tweak useful at G?
There was a problem hiding this comment.
I am pretty sure this is optimization is necessary as we can have large type hierarchies. I have asked internally for more background, but I expect this optimization to be necessary to not have a large performance impact for existing applications.
There was a problem hiding this comment.
Adding perf tweaks "on hunch", without evidence is not good for the codebase (growing complexity :-). I don't mind this change but I'd love to get your assurance that this is useful.
You have the approval so ship this change at will!
There was a problem hiding this comment.
I will double check before merging. I haven't had time during the Christmas vacation to do this work. I probably get to this late next week.
They won't compile for Android users
|
I have done an initial pass at integrating this internally with good results! We are doing additional verification to confirm everything is working as expected, but thus far it is working out. |
|
Hey, this is a fantastic feature and I'd love for this to make it in. Can I gently nudge the and reviewers to finish this up? |
|
I still have to integrate this in the Google monorepository and figure out whether it is working correctly. Sadly I haven't had the time to do so. |
|
#2026 is another instance where we could benefit from |
src/main/java/org/mockito/internal/configuration/DefaultDoNotMockEnforcer.java
Outdated
Show resolved
Hide resolved
Codecov Report
@@ Coverage Diff @@
## main #1833 +/- ##
============================================
+ Coverage 86.46% 87.10% +0.64%
+ Complexity 2763 2610 -153
============================================
Files 318 316 -2
Lines 8304 6793 -1511
Branches 1016 848 -168
============================================
- Hits 7180 5917 -1263
+ Misses 852 675 -177
+ Partials 272 201 -71
Continue to review full report at Codecov.
|
|
Thanks everyone for your input and feedback! I will be publishing Mockito 4.1.0 that includes this feature 🎉 |
I forgot to add the DoNotMock feature to the main JavaDoc in #1833
I forgot to add the DoNotMock feature to the main JavaDoc in #1833
Mocking types that users not own 1 or are severely complicating test
logic 2 leads to brittle or wrong tests. In particular, the
StackOverflow answer is wrong, as the contract of java.util.Map is
violated. When a new key is added to the Map, the stubbed return would be wrong.
In Google we have used the DoNotMock annotation via ErrorProne 3
to annotate these types, as well as an internal list of types that can't
be mocked (this includes several java.util types). We are using a custom
Mockmaker to enforce this on run-time.
Based on our successful experience with DoNotMock (we have seen a large
reduction in bad/broken tests for types involved), we are proposing to
open source this into Mockito itself.
The DoNotMock annotation can be added to any type, e.g. classes and
interfaces. If, in the type hierarchy of the class-to-be-mocked, there
is a type that is annotated with DoNotMock, Mockito will throw a
DoNotMockException.
This would help preventing issues such as #1827 and #1734 which is
in-line with the guidance on our wiki 1. A follow-up change would
allow us to define external types (like the java.util types) that can't
be mocked. (We can't add the annotation to the types, as they live in the
JDK instead.)