TeamCity service message should have assembly name as a part of test name.

[NikolayPianikov]

#671

For example ##teamcity[testStarted name='NUnit.Engine.Tests.TestEngineResultTests.CanCreateFromXmlString' captureStandardOutput='true' flowId='0-1111']

teamcity[testFinished name='NUnit.Engine.Tests.TestEngineResultTests.CanCreateFromXmlString' duration='0.001' flowId='0-1111']

should be:

teamcity[testStarted name='nunit.core.engine.tests.dll: NUnit.Engine.Tests.TestEngineResultTests.CanCreateFromXmlString' captureStandardOutput='true' flowId='0-1111']

teamcity[testFinished name='nunit.core.engine.tests.dll: NUnit.Engine.Tests.TestEngineResultTests.CanCreateFromXmlString' duration='0.001' flowId='0-1111']

It is needed when tests have the same name and another original assembly

[CharliePoole]

I feel like we are thrashing on this issue. At @chris-smith-zocdoc 's request, we created the flowId, which is unique across all assemblies, to resolve the same problem. If it's not working, we should fix it of course but maybe we need to have a better understanding of the purpose of each of the parameters before making any more changes.

We probably need to have our teamcity build working before doing this as well.

[chris-smith-zocdoc]

I made some progress with the TC build but did find one new issue #670

This particular issue

when tests have the same name and another original assembly

was actually brought up in #590 (comment) but haven't been motivated to resolve it. I feel that @NikolayPianikov 's suggestion is the correct solution though.

[NikolayPianikov]

The "flowId" is just "id of thread" and should be unique if tests are processed in parallel. Thus the "flowId" doesn't help when user has several different assemblies which contain some test methods with the same names (including class's name and namespace). User has several results for his tests and have no ability to collate specific result and corresponding test.
I tried to support backward compatibility. All previous nunit.framework versions should work with new NUnit 3 console. I added attribute "assembly" to xml report. This assembly is used to specify additional prefix of test name in the TeamCity service message. See pull request #671

I would like to add "start suite" and "stop suite" service messages to handle cases when the names of assemblies are the same (it is actual for multi platform build for example). Are there any objections?

[chris-smith-zocdoc]

I would like to add "start suite" and "stop suite" service messages to handle cases when the names of assemblies are the same (it is actual for multi platform build for example). Are there any objections?

The "start suite" and "stop suite" messages were removed in #590 because they interfered with the parallel test execution. Is it possible to use them and run tests in parallel? Most of the discussion around this happened in the google group https://groups.google.com/forum/#!topic/nunit-discuss/X2-LDrsxXXc

[NikolayPianikov]

As I understood from the discussion we can start tests within some suite using different TestWorkers and we have no way to assign correct flowId for test suites. We could use following TC service message ##teamcity[flow Started flowId=''] and ##teamcity[flow Started flowId='' parent=''] to show tests' results correctly in the TeamCity. These messages allow us to specify the tree of test items (test, test suite) properly. I can do it in the scope of another task may be.

[NikolayPianikov]

I've checked ##teamcity[flowStarted ...] and it works fine.
But I think this change is not important now - TC uses his own grouping on the client.
And what do you think about "TeamCity service message should have assembly name as a part of test name"?

[chris-smith-zocdoc]

TeamCity service message should have assembly name as a part of test name

Yes I think this is a good change. It allows users to filter by assembly which is nice.

[CharliePoole]

Maybe we need a page in one of our wikis about exactly what messages we send (or want to send) for each teamcity event. If someone wants to volunteer to do this we can give you access to create it.

Also, I am beginning to think that our flowid goes too far. We created a flowid that is actually unique for every test and suite. This is probably not serving the purpose for which flowid was intended. Maybe we should try to actually provide what TeamCity asks for: one flowid per thread of execution.

[NikolayPianikov]

I can fill this page, but it would be nice to make review after

TC has the bug, related to our situation - unique flowId per test, but I am going to fix it in the TC. FlowId per test is acceptable solution from my point of view. Solution "one flower per thread of execution" - will be better (but it is not so critical), we could schedule this task and I could make.

[CharliePoole]

I created a stub page and gave you access to edit the wiki...

https://github.com/nunit/dev/wiki/TeamCity-Service-Messages

I put it under Specifications in the development wiki for now. When we implement, it can become a Technical Note in the nunit wiki.

The unique id per test was intended as a workaround, which we hoped would solve all problems of duplication. That isn't so because TC only uses the flowid for a specific purpose and not for actually identifying unique tests. We could consider using this unique id as a prefix or suffix to every test name, rather than the assembly name. It's a question of what looks best in the reports, I suppose.

Of course, users don't need to use this feature if they prefer not to. They could instead use the report that is generated from the NUnit XML output.

[CharliePoole]

In writing up the service messages and how we use them, I suggest you use links to the TeamCity documentation for each one.

[NikolayPianikov]

OK, thanks @CharliePoole, I will look