Injection problems, 1.13 and newer messes with TestNG's reporting

[chesshir]

I have two bugs to report on here. I have a sample project that demonstrates both. Please let me know where to send it.

1. Both JMockit and TestNG have the ability to do parameter Injection. TestNG does this through a test function's association to a data provider. Used separately, both JMockit and TestNG work fine. However, when you try to mock some parameters to a test function and hook up a Data Provider to inject the rest of them, TestNG has no way of knowing that it doesn't need to provide those extra parameters. I tried passing a mocked parameter to the data provider and passing it on to the Test function, but JMockit does not recognize the Mocked attribute for parameters to any functions that are not marked with the Test attribute. I would like to either see this enabled or, better yet, JMockit find the capability to tell TestNG that it has got certain parameters covered.

FYI, I turned in this bug to the TestNG folks as well, but it does not appear that they will be able to help. You can view that ticket here: cbeust/testng#600 (comment)

2. When a TestNG Data Provider is hooked up to a test function, Eclipse's TestNG plugin will report on each test case the data provider hands to the function. Including JMockit in the project has no effect on this for versions 1.12 and earlier. However, starting with 1.13, when you merely include JMockit in the project, TestNG's ability to report on individual test cases is broken. All indications are that the test still runs. However, Eclipse only shows the test function itself, not the individual test cases handed to it by the Data Provider. My team has a custom reporting mechanism that exhibits the same problem. (That mechanism cannot be included in the sample, though.)

[chesshir]

I found the user group. I will send the sample there.

[rliesenfeld]

Item 1 would be an enhancement, not a bug. But before considering it, I would need to understand what the real-world uses are.

Item 2 seems the same as issue #121, which was already fixed for the next release.

[chesshir]

For item 2: Yes, that does appear to be a duplicate. I look forward to seeing 1.15 come out so I can verify it is fixed.

For item1: My specific use case came from mocking the BufferedWriter class with a mocked field. I found that when I had that mocked field present, TestNG would not report the test results for any of the tests in that Test class, and so my build was getting a false success. In an attempt to circumvent that problem I tried making it a mocked parameter to the only function that actually needed it mocked. My hope was that the decreased scope would keep it from mocking whatever instance that TestNG needed to perform its work. Unfortunately, that function was also hooked up to a Data Provider, and I could not find a way to restrict the Mocked range on the BufferedWriter class any other way. I ended up having to let the tested function write out its contents to a real file and then verify the contents of that file.
If, as Cedric said in his reply to my TestNG post, there can be found no way to combine TestNG and JMockit injection into a single test function, it would be sufficient to be able to Mock a data provider field, which could then be passed onto the Test function as a regular parameter. Of course that might provide a further gotcha considering that the mocked range on the class could go out of scope at the end of the Data Provider function before it get's passed on to the Test function. On the other hand, if one put the Mocked tag on the parameter in the Test function as well, the Mocked scope could be resumed while at the same time satisfying TestNG's need for the Data Provider to provide all of the parameters.
Of course, it would look cleaner if we could just allow TestNG and JMockit to inject parameters into the same function at the same time. And getting TestNG to let you inject a parameter to the Data Provider may be just as challenging.
But I know those are just an outsider's theories. :)

[rliesenfeld]

I will look into the conflict with TestNG when using "@Mocked BufferedWriter" mock field; ideally, it should work. And if it can't be made to work reliably, then JMockit shoud forbid the full mocking of BufferedWriter; it's probably not a good idea anyway to mock such classes, as such code is an implementation detail that you don't want tests to depend on.

Support for mock parameters in test methods with data providers could be added to JMockit, I imagine. I won't be doing it any time soon, though, as there is plenty of more important things to work on for now. I can accept a PR, though.