Add support for VSTS logging in SpecRun.exe

[ChristopherHaws]

Hello,

Currently, specrun.exe has console output for TFS, VS2010 - VS2015, and TeamCity. It would be nice to have a VSTS option which would log using VSTS's format which is:

##vso[task.loglevel]Message

So, an example would be:

##vso[task.debug]Hello World!

Thanks! :)

[SabotageAndi]

Is there a special reason why you are executing the tests via specrun.exe on TFS?
You can use the Test Step with the SpecFlow+Runner Test Adapter (which is also used in VS). Then the logging is done via the VSTest infrastructure.

Documentation to setup is here: http://specflow.org/plus/documentation/SpecFlowPlus-and-Build-Servers/

[ChristopherHaws]

We would prefer to use VSTest, but our test suit has random failures, which is common with selenium UI automation testing, so we have our tests setup to rerun 2 times if there is a failure. With SpecRun.exe we can get the exit code (430 - RandomlyFailed) and choose to allow this to still be a passed test run, but VSTest.console.exe returns exit code 1 if there are any failures.

If you know of a way around this, we would love to know it. We ended up having to create a custom SpecRun report that mimics a TRX file because we couldn't get this working with VSTest.

[SabotageAndi]

I understand and it makes sense. Nice that you were able to mimic the TRX file with a report template!

I know, that we have in our backlog the feature request, that the build is passed, if the retries were successful.
That should also work then in your situation, or?

I will search for it next week and have a quick look if it is possible to do this with VSTest.

[ChristopherHaws]

That would be fantastic! It would really make our lives a lot easier if it was handled in the test adapter.

Thanks! :)

[SabotageAndi]

So it took a little bit longer as expected, but with the today released pre-release of SpecFlow+Runner (1.7.0-rc001), we added this feature.
Be aware that this is currently to be seen as experimental and could be changed or removed from the runner.
It is also only available when you execute the tests via VSTest (VisualStudio, TFS/VSTS or vstest.console.exe).

We added a new configuration for VSTest only to unify the result of retries depending on a passrate.

Example config:

        <?xml version="1.0" encoding="utf-8"?>
        <TestProfile xmlns="http://www.specrun.com/schemas/2011/09/TestProfile">
            <Execution retryFor="Failing" retryCount="1" />
            <VSTest TestRetryResults="Unified" passRate="100" />
            <TestAssemblyPaths>
                <TestAssemblyPath>SpecRun.TestProject.dll</TestAssemblyPath>
            </TestAssemblyPaths>
        </TestProfile>

Documentation:
VsTest
This section determines the behaviour when using SpecFlow+ Runner in conjunction with VS Test.

TestRetryResults: By default, a test that initially fails is treated as having failed, even if the retries are successful. This can result in test execution results that show the same test as both failing (the initial test) and passing (the subsequent retries). This is particularly common when testing web applications if the initial test times out. In some cases, you may want to determine that a test should be treated as passing, even if one or other of the tests fails, allowing you to discount tests that time out.
The following options are available:
Separate (default): When retrying tests, separate test results are output for each execution of the test: the initial (failing) execution and all retries.
Unified: A single unified result is output for the test. If the percentage of successful tests (the initial test + all retries) exceeds the threshold (passRate), the test is treated as passing.

passRate: Determines the percentage threshold (default = 100%) used to determine when a test should be treated as passing when using the Unified option. For example, if the passRate is set to 50%, the test is only counted as passing if at least half the tests pass.

Please give us feedback on this feature if it solves your issues.

[SabotageAndi]

@ChristopherHaws Did you had any chance to have a look at this?

[ChristopherHaws]

@SabotageAndi I will take a look at this right now. Thanks for the update! :)

[ChristopherHaws]

@SabotageAndi Would it be possible to have a TestRetryResults option similar to Unified that is for AtleastOnce? For example, with the current setup, if our retryCount was set to 3, we would need to know to set the passRate to 33.333 in order for this to work.

In our case, we only care if the test passes at least one time.

[ChristopherHaws]

@SabotageAndi I am receiving the following error when trying to run the test:

[2017-11-03 3:03:23 PM Informational] TechTalk.SpecRun.Framework.SpecRunConfigurationException: Invalid configuration file C:\Users\chaws\Source\Repos\SpecRunTestRetry\SpecRunTestRetry\bin\Debug\Default.srprofile:There is an error in XML document (7, 4). -> Unknown configuration node: VSTest ---> System.InvalidOperationException: There is an error in XML document (7, 4). ---> TechTalk.SpecRun.Framework.SpecRunConfigurationException: Unknown configuration node: VSTest
   at TechTalk.SpecRun.Framework.Configuration.TestProfileLoader.<>c.<LoadProfileWithDefaultNamespace>b__1_2(Object sender, XmlNodeEventArgs args)
   at System.Xml.Serialization.XmlSerializationReader.UnknownNode(XmlNode unknownNode, Object o, String qnames)
   at System.Xml.Serialization.XmlSerializationReader.UnknownNode(Object o, String qnames)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderTestProfile.Read30_TestProfile(Boolean isNullable, Boolean checkType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderTestProfile.Read31_TestProfile()
   --- End of inner exception stack trace ---
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
   at System.Xml.Serialization.XmlSerializer.Deserialize(TextReader textReader)
   at TechTalk.SpecRun.Framework.Configuration.TestProfileLoader.LoadProfileWithDefaultNamespace(String filePath, String defaultNamespace)
   --- End of inner exception stack trace ---
   at TechTalk.SpecRun.Framework.Configuration.TestProfileLoader.LoadProfileWithDefaultNamespace(String filePath, String defaultNamespace)
   at TechTalk.SpecRun.Framework.Configuration.TestProfileLoader.LoadTestProfileFromFile(String filePath)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
   at TechTalk.SpecRun.Framework.Configuration.TestProfileLoader.LoadTestProfileFromFirstExistingFile(IEnumerable`1 filePaths)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.Settings.ProfileProvider.GetTestProfile(IDiscoveryContext discoveryContext, String source, TestingContext testingContext)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.TestingContext.LoadTestProfile(String source)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.SpecRunTestExecutor.RunTests(String source, IRunContext runContext, IFrameworkHandle frameworkHandle, FilterItem filter, TestingContext testingContext)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.SpecRunTestExecutor.RunTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle)
[2017-11-03 3:03:23 PM Error] An exception occurred while invoking executor 'executor://specrun/executorV1.7.0': Invalid configuration file C:\Users\chaws\Source\Repos\SpecRunTestRetry\SpecRunTestRetry\bin\Debug\Default.srprofile:There is an error in XML document (7, 4). -> Unknown configuration node: VSTest
There is an error in XML document (7, 4).
Unknown configuration node: VSTest

If I remove the VSTest element it runs without issue.

Packages.config

<packages>
  <package id="FluentAssertions" version="4.19.4" targetFramework="net461" />
  <package id="Newtonsoft.Json" version="9.0.1" targetFramework="net461" />
  <package id="SpecFlow" version="2.2.1" targetFramework="net461" />
  <package id="SpecRun.Runner" version="1.7.0-rc001" targetFramework="net461" />
  <package id="SpecRun.SpecFlow" version="1.7.0-rc001" targetFramework="net461" />
  <package id="SpecRun.SpecFlow.2-2-0" version="1.7.0-rc001" targetFramework="net461" />
</packages>

Default.srprofile

<?xml version="1.0" encoding="utf-8"?>
<TestProfile xmlns="http://www.specrun.com/schemas/2011/09/TestProfile">
  <Settings projectName="SpecRunTestRetry" projectId="{D29DEB5E-F080-434A-9E56-9DA3DFEEC25F}" />
  <Execution retryFor="Failing" retryCount="3" />
  <VSTest TestRetryResults="Unified" passRate="100" />

  <TestAssemblyPaths>
    <TestAssemblyPath>SpecRunTestRetry.dll</TestAssemblyPath>
  </TestAssemblyPaths>

</TestProfile>

[SabotageAndi]

@ChristopherHaws There were some issues with 1.7-rc001 and this feature. They are fixed with the 1.7-rc002 release.

[SabotageAndi]

SpecFlow+Runner 1.7.1 is out with fixes for the Unified VSTest Test result mode.

[CleytonGoncalves]

@SabotageAndi I am receiving the following error when trying to run with <VSTest TestRetryResults="Unified" passRateAbsolute="1" />.

Error

[09/05/2018 10:34:17 Informational] TechTalk.SpecRun.Framework.SpecRunConfigurationException: Invalid configuration: Erro no documento XML (5, 29). -> Unknown configuration node: TestRetryResults ---> System.InvalidOperationException: Erro no documento XML (5, 29). ---> TechTalk.SpecRun.Framework.SpecRunConfigurationException: Unknown configuration node: TestRetryResults
   em TechTalk.SpecRun.Framework.Configuration.TestProfileLoader.<>c.<LoadProfileWithDefaultNamespace>b__8_2(Object sender, XmlNodeEventArgs args)
   em System.Xml.Serialization.XmlSerializationReader.UnknownNode(XmlNode unknownNode, Object o, String qnames)
   em System.Xml.Serialization.XmlSerializationReader.UnknownNode(Object o, String qnames)
   em Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderTestProfile.Read16_VSTestSettings(Boolean isNullable, Boolean checkType)
   em Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderTestProfile.Read32_TestProfile(Boolean isNullable, Boolean checkType)
   em Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderTestProfile.Read33_TestProfile()
   --- Fim do rastreamento de pilha de exceções internas ---
   em System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
   em System.Xml.Serialization.XmlSerializer.Deserialize(TextReader textReader)
   em TechTalk.SpecRun.Framework.Configuration.TestProfileLoader.LoadProfileWithDefaultNamespace(String content, String defaultNamespace)
   --- Fim do rastreamento de pilha de exceções internas ---
   em TechTalk.SpecRun.Framework.Configuration.TestProfileLoader.LoadProfileWithDefaultNamespace(String content, String defaultNamespace)
   em TechTalk.SpecRun.Framework.Configuration.TestProfileLoader.LoadProfileWithDefaultNamespace(TextReader reader, String fileName)
   em TechTalk.SpecRun.Framework.Configuration.TestProfileLoader.LoadTestProfileFromFile(String filePath)
   em System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   em System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
   em TechTalk.SpecRun.Framework.Configuration.TestProfileLoader.LoadTestProfileFromFirstExistingFile(IEnumerable`1 filePaths)
   em TechTalk.SpecRun.VisualStudio.TestAdapter.Settings.ProfileProvider.GetTestProfile(IDiscoveryContext discoveryContext, String source, TestingContext testingContext)
   em TechTalk.SpecRun.VisualStudio.TestAdapter.TestingContext.LoadTestProfile(String source)
   em TechTalk.SpecRun.VisualStudio.TestAdapter.SpecRunTestDiscoverer.DiscoverTests(String source, IDiscoveryContext discoveryContext, IMessageLogger logger, ITestCaseDiscoverySink discoverySink)
   em TechTalk.SpecRun.VisualStudio.TestAdapter.SpecRunTestDiscoverer.DiscoverTests(IEnumerable`1 sources, IDiscoveryContext discoveryContext, IMessageLogger logger, ITestCaseDiscoverySink discoverySink)
[09/05/2018 10:34:17 Error] Ocorreu uma exceção enquanto o discoverer de testes 'SpecRunTestDiscoverer' estava carregando testes. Exceção: Invalid configuration: Erro no documento XML (5, 29). -> Unknown configuration node: TestRetryResults
[09/05/2018 10:34:17 Informational] ========== Discover test finished: 0 found (0:00:01,098362) ==========

Default.srprofile

<?xml version="1.0" encoding="utf-8"?>
<TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5">
  <Settings projectName="TesteAutomatizado" projectId="{a896309f-d68e-4381-bf1d-0c42166698d8}" />
  <Execution stopAfterFailures="10" testThreadCount="1" testSchedulingMode="Sequential" />
  <VSTest testRetryResults="Unified" passRateAbsolute="1" />
  <!-- For collecting by a SpecRun server update and enable the following element. For using the 
      collected statistics, set testSchedulingMode="Adaptive" attribute on the <Execution> element.
    <Server serverUrl="http://specrunserver:6365" publishResults="true" />
  -->
  <TestAssemblyPaths>
    <TestAssemblyPath>TesteAutomatizado.dll</TestAssemblyPath>
  </TestAssemblyPaths>
  <DeploymentTransformation>
    <Steps>
      <!-- sample config transform to change the connection string-->
      <!--<ConfigFileTransformation configFile="App.config">
        <Transformation>
          <![CDATA[<?xml version="1.0" encoding="utf-8"?>
							<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
                <connectionStrings>
                  <add name="MyDatabase" connectionString="Data Source=.;Initial Catalog=MyDatabaseForTesting;Integrated Security=True" 
                       xdt:Locator="Match(name)" xdt:Transform="SetAttributes(connectionString)" />
                </connectionStrings>
							</configuration>
						]]>
        </Transformation>
      </ConfigFileTransformation>-->
    </Steps>
  </DeploymentTransformation>
</TestProfile>

Visual Studio Tooltip

[SabotageAndi]

Are you using the latest version?

[CleytonGoncalves]

I'm using v1.7.1.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.