-
Notifications
You must be signed in to change notification settings - Fork 728
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Erroneous TeamCity service messages with multiple test assemblies and (default) parallel execution #1745
Comments
Please indicate the version of the console runner you are using. |
Issue observed with 3.4.1 as well as 3.5.0-dev-03126. |
@CharliePoole I will try to reproduce it in integration tests |
Note that all tests seem to be executed and listed, but the suites (assemblies) have unpredictable representation in the report. This produces structural differences in the reported test results when picked up by TeamCity. |
The problem using |
@einert could you attach TC build log file, could you reproduce it from the command line and attach the console ouput too. @CharliePoole I found another problem while reproducing it. I've received "Could not load file or assembly 'nunit.framework' or one of its dependencies. The system cannot find the file specified." message when I added "--domain=None" to the command line. |
See issue #1741. The --domain=None option is an obscure one and should really only be used in special cases, where some code being tested can only run in a primary AppDomain. It requires all files, including NUnit's, to be accessible in the AppDomain. Is there a reason to use it here? |
@CharliePoole I've added integration tests for all variants including --domain=None. |
@NikolayPianikov I'm afraid I can't easily attach the full TC build log verbatim due to IP restrictions. However, I can show the structure here before and after restricting agents by anonymizing a little. Original build, only running with --teamcity argument:
After setting --agent=1, assemblies/suites are run consecutively and identified as "started" and "finished", which manifests like this when captured by TeamCity:
|
@NikolayPianikov Yes. That's what I would expect. |
@einert could you attach the output including TC service messages for the case 1. |
@NikolayPianikov This is not necessarily identical to case #1, but is a fresh run with the same symptomatic behavior (no agent restriction):
As you can see:
|
@einert what version of nunit.framework.dll do you use? |
@NikolayPianikov Nailed it! Turns out that the project in question had had a partial tool update done. All test assemblies were built against nunit.framework.dll 2.6.1. Tests were run with nunit3-console 3.4.1. After upgrading all references, things are looking much better. |
Issue moved to nunit/teamcity-event-listener #14 via ZenHub |
When running nunit3-console.exe for a list of several (six) test assemblies and with the
--teamcity
flag, like so:nunit3-console.exe --teamcity first.test.dll second.test.dll ... last.test.dll
, the service messages produced are incomplete. Specifically, the output does not contain more than onetestSuiteStarted
message, and onetestSuiteFinished
message, which does not match the start one. Which suite/flow is reported seems to be arbitrary.One consequence of this is that the report in TeamCity is different from run to run in terms of suite placement, even if nothing has changed. This can really mess with test report consistency if some assembly and/or test names are very long, considering the 255 character limit on the combined suite + assembly + test name.
The expected output is a full set of matching start/finish messages, one for each assembly, each with their own
flowId
.Adding '--agents=1' yields the expected (correct) output.
Incidentally, the first attempt, using the
--inprocess
argument with--teamcity
, did not really work out, but produced:The text was updated successfully, but these errors were encountered: