Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BeforeTestRun hook is executed multiple times when running tests in parallel using SpecFlow+ Runner with SharedAppDomain #1917

Closed
CaptnBigShot opened this issue Mar 19, 2020 · 5 comments

Comments

@CaptnBigShot
Copy link

SpecFlow version: 3.1
SpecFlow+ version: 3.1
Test runner: Specflow+ Runner
.NET framework version: 4.5
Visual Studio version: 16.5.0

Attached reproducible project: SpecflowTest.zip

SpecFlow NuGet Packages:

  <package id="SpecFlow" version="3.1.89" targetFramework="net45" />
  <package id="SpecFlow.Tools.MsBuild.Generation" version="3.1.89" targetFramework="net45" />
  <package id="SpecRun.Runner" version="3.1.53" targetFramework="net45" />
  <package id="SpecRun.SpecFlow" version="3.1.53" targetFramework="net45" />
  <package id="SpecRun.SpecFlow.3-1-0" version="3.1.53" targetFramework="net45" />

srprofile:

<?xml version="1.0" encoding="utf-8"?>
<TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5">
  <Settings name="Default" projectName="SpecflowTest" projectId="{f59c1919-b01f-4cd2-bcb7-71a4b6f624d2}" />
  <Execution stopAfterFailures="0" testThreadCount="3" retryCount="0" />
  <Environment testThreadIsolation="SharedAppDomain" />
  <TestAssemblyPaths>
    <TestAssemblyPath>SpecflowTest.dll</TestAssemblyPath>
  </TestAssemblyPaths>
</TestProfile>

Description
According to the documentation, when testThreadIsolation is set to 'SharedAppDomain', a BeforeTestRun hook is only to be executed once. This would allow for running tests in parallel without having to run the BeforeTestRun multiple times.

However, when running tests in parallel using multiple threads with testThreadIsolation set to 'SharedAppDomain', the BeforeTestRun hook is executed multiple times. It is executed on one thread multiple times; specifically, the number of threads specified by testThreadCount. This occurs when the tests are run via VSTest or SpecRun.exe.

The Hooks.cs file in the attached project contains a BeforeTestRun hook, which prints "[BeforeTestRun]" to the console.

Below is an excerpt from the log file produced after running the tests:

2020-03-19T11:34:46.1658837-04:00:Thread#1:T:-> Using default config
2020-03-19T11:34:46.6086923-04:00:Thread#1:T:[BeforeTestRun]
2020-03-19T11:34:46.6096901-04:00:Thread#1:T:[BeforeTestRun]
2020-03-19T11:34:46.6096901-04:00:Thread#1:T:[BeforeTestRun]
2020-03-19T11:34:46.6146768-04:00:Thread#2: Test #5/0 is started on thread #2
2020-03-19T11:34:46.6156843-04:00:Thread#0: Test #4/0 is started on thread #0
2020-03-19T11:34:46.6156843-04:00:Thread#1: Test #6/0 is started on thread #1
@SabotageAndi
Copy link
Contributor

Interesting. Thanks for reporting this. We will have a look at it.

@EmilyRocks
Copy link

Hi! I am experiencing the same problem but with a different version of SpecFlow+ Runner. Is there a version where the BeforeTestRun hook executes only once for multithreaded test execution or some other combination of settings I can use as a workaround in my srprofile to get BeforeTestRun to execute once for multiple threads?

@SabotageAndi
Copy link
Contributor

@EmilyRocks it depends on your test thread isolation. If it is SharedAppDomain, the hooks should only be called once.
In any other mode, it will get executed once per thread. More details here: https://docs.specflow.org/projects/specflow/en/latest/Execution/Parallel-Execution.html#id3

@SabotageAndi
Copy link
Contributor

As we stopped maintaining the SpecFlow+ Runner (https://specflow.org/using-specflow/the-retirement-of-specflow-runner/) I am closing this issue.

@github-actions
Copy link

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants