Too many tests are run (and some fail) when test methods exist in more than 1 test module

[ThunderFrame]

Using the Test Explorer commands in Excel 2016 32-bit, I added a test module, and a single test method, to an empty Excel 2016 project. And then, again using the Test explorer menu commands, added another test module and single test method. I then edited the name of the 2nd test module's test method name to TestMethod2 for clarity (although with scoping, I shouldn't need to). So, I have:

TestModule1.TestMethod1
TestModule2.TestMethod2

All other module/test initialization/Cleanup methods have the default name. The Assert objects are each defined using Late-binding.

When I executed the tests, I got VBE errors because the Assert object was not set, so I added debug.print statements to each method.

This is the output in the debug window (with comments added afterwards):

ModuleInitialize (TestModule1)
TestInitialize (TestModule1)
TestMethod1 (TestModule1)
TestCleanup (TestModule1)
TestInitialize (TestModule1)
TestMethod2 (TestModule2) - Failed  'This method belongs to TestModule2!
TestCleanup (TestModule1)
ModuleCleanup (TestModule1)
ModuleInitialize (TestModule2)
TestInitialize (TestModule2)
TestMethod1 (TestModule1) - Failed  'This method belongs to TestModule1!
TestCleanup (TestModule2)
TestInitialize (TestModule2)
TestMethod2 (TestModule2) - Failed  'This method belongs to TestModule2!
TestCleanup (TestModule2)
ModuleCleanup (TestModule2)

As you can see, Rubberduck had treated TestMethod2 as if it belonged to TestModule1 and when it tried to execute it, the Module/Test had not been initialized. Later, when it tries to run methods in TestModule2, it erroneously tried to run TestMethod1 (which fails), and then when it tries to run TestMethod2 (as it should, for TestModule2), it fails unexpectedly (although I think it's because the Assert object has been destroyed by me Ending the code (when TestMethod1 failed).

Furthermore, Rubberduck tried to call each Test Method exactly twice. And it does so whether the Reparse state results in errors, or results in success.

Unfortunately, I can't repeat this behavior all of the time, but it is reproducible, and in my experience, once you get the errors, you will repeatedly get the errors until you edit the module and trigger a reparse. I suspect that this is a parser error under certain parse results. That is, parse results do not appear to be deterministic / consistent.

Here are the module contents:

Attribute VB_Name = "TestModule1"
Option Explicit
Option Private Module

'@TestModule
Private Assert As Object

'@ModuleInitialize
Public Sub ModuleInitialize()
    Debug.Print "ModuleInitialize (TestModule1)"
    'this method runs once per module.
Set Assert = CreateObject("Rubberduck.AssertClass")

End Sub

'@ModuleCleanup
Public Sub ModuleCleanup()
    Debug.Print "ModuleCleanup (TestModule1)"
    'this method runs once per module.
End Sub

'@TestInitialize
Public Sub TestInitialize()
    Debug.Print "TestInitialize (TestModule1)"
    'this method runs before every test in the module.
End Sub

'@TestCleanup
Public Sub TestCleanup()
    Debug.Print "TestCleanup (TestModule1)"
    'this method runs after every test in the module.
End Sub

'@TestMethod
Public Sub TestMethod1() 'TODO Rename test
    On Error GoTo TestFail
    Debug.Print "TestMethod1 (TestModule1)"

    'Arrange:

    'Act:

    'Assert:
    Assert.Inconclusive

TestExit:
    Exit Sub
TestFail:
    Debug.Print "Failed"
    Assert.Fail "Test raised an error: #" & Err.Number & " - " & Err.Description
End Sub

Attribute VB_Name = "TestModule2"
Option Explicit
Option Private Module

'@TestModule
Private Assert As Object

'@ModuleInitialize
Public Sub ModuleInitialize()
    Debug.Print "ModuleInitialize (TestModule2)"
    'this method runs once per module.
Set Assert = CreateObject("Rubberduck.AssertClass")

End Sub

'@ModuleCleanup
Public Sub ModuleCleanup()
    Debug.Print "ModuleCleanup (TestModule2)"
    'this method runs once per module.
End Sub

'@TestInitialize
Public Sub TestInitialize()
    Debug.Print "TestInitialize (TestModule2)"
    'this method runs before every test in the module.
End Sub

'@TestCleanup
Public Sub TestCleanup()
    Debug.Print "TestCleanup (TestModule2)"
    'this method runs after every test in the module.
End Sub

'@TestMethod
Public Sub TestMethod2() 'TODO Rename test
    On Error GoTo TestFail
    Debug.Print "TestMethod2 (TestModule2)"
    'Arrange:

    'Act:

    'Assert:
    Assert.Inconclusive

TestExit:
    Exit Sub
TestFail:
    Debug.Print "Failed"
    Assert.Fail "Test raised an error: #" & Err.Number & " - " & Err.Description
End Sub

[ThunderFrame]

Well, I decided that me Ending the code when it was breaking wasn't helping, because it was destroying the Assert objects unnecessarily, so I commented out the line:

Assert.Fail "Test raised an error: #" & Err.Number & " - " & Err.Description

And then I get this output (indentation added for clarity):

ModuleInitialize (TestModule2)
  TestInitialize (TestModule2)
    TestMethod2 (TestModule2)
  TestCleanup (TestModule2)
  TestInitialize (TestModule2)
    TestMethod1 (TestModule1) - Failed
  TestCleanup (TestModule2)
ModuleCleanup (TestModule2)

ModuleInitialize (TestModule1)
  TestInitialize (TestModule1)
    TestMethod2 (TestModule2)
  TestCleanup (TestModule1)
  TestInitialize (TestModule1)
    TestMethod1 (TestModule1)
  TestCleanup (TestModule1)
ModuleCleanup (TestModule1)

So the TestMethods are still being called multiple times, but only the method call for which module intialization hasn't occurred yet is failing. And, I realized that the results are reproducable, but only if you reset the VBA project between each Run All Tests execution, because otherwise the Assert objects are set on the first run, and remain set on subsequent runs.

So, I added a third TestModule along with TestMethod3, and I get these results (indentation added for clarity). Notice that 3 tests are called for each module. This smells like a Linq bug.

ModuleInitialize (TestModule2)
  TestInitialize (TestModule2)
    TestMethod2 (TestModule2)
  TestCleanup (TestModule2)
  TestInitialize (TestModule2)
    TestMethod1 (TestModule1) - Failed
  TestCleanup (TestModule2)
  TestInitialize (TestModule2)
    TestMethod3 (TestModule3) -Failed
  TestCleanup (TestModule2)
ModuleCleanup (TestModule2)

ModuleInitialize (TestModule1)
  TestInitialize (TestModule1)
    TestMethod2 (TestModule2)
  TestCleanup (TestModule1)
  TestInitialize (TestModule1)
    TestMethod1 (TestModule1)
  TestCleanup (TestModule1)
  TestInitialize (TestModule1)
    TestMethod3 (TestModule3) - Failed
  TestCleanup (TestModule1)
ModuleCleanup (TestModule1)

ModuleInitialize (TestModule3)
  TestInitialize (TestModule3)
    TestMethod2 (TestModule2)
  TestCleanup (TestModule3)
  TestInitialize (TestModule3)
    TestMethod1 (TestModule1)
  TestCleanup (TestModule3)
  TestInitialize (TestModule3)
    TestMethod3 (TestModule3)
  TestCleanup (TestModule3)
ModuleCleanup (TestModule3)