Question: why not add IsTestProject to project property when create a test project?

[WeihanLi]

As the title, why not add <IsTestProject>true</IsTestProject> to the test project,
it seemed the ci platform like azure devops and travis will not run the test when no <IsTestProject>true</IsTestProject> specified

[dasMulli]

The IsTestProject property is set when you reference the test SDK. Specifying it for the templates that include the test SDK would be redundant..
What exactly is your use case?

[WeihanLi]

when I create the project in vs, it may like this https://github.com/WeihanLi/WeihanLi.Npoi/blob/b03d4f54d75ee40038cf52652dedad2f0723a32e/test/WeihanLi.Npoi.Test/WeihanLi.Npoi.Test.csproj, you can not run your tests on travis or Azure Devops unless add IsTestProject

  
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.0</TargetFramework>
    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.2.0" />
    <PackageReference Include="xunit" Version="2.4.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.0" />
    <PackageReference Include="coverlet.collector" Version="1.0.1" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\..\src\WeihanLi.Npoi\WeihanLi.Npoi.csproj" />
  </ItemGroup>

</Project>

[dasMulli]

Is that Tests in VS or other pieces of software? I’m not sure this should be in templates if it might be a bug in other software looking for this in XML (which they probably shouldn’t do)

[WeihanLi]

it works well on my local develop environment (windows 10), but not work on Azure Devops(event Windows Agent) or travis

sample build history here:

• with <IsTestProject>true</IsTestProject>:

  https://weihanli.visualstudio.com/Pipelines/_build/results?buildId=1393&view=logs&j=12f1170f-54f2-53f3-20dd-22fc7dff55f9

  

• without IsTestProject:

  https://weihanli.visualstudio.com/Pipelines/_build/results?buildId=1388&view=logs&j=12f1170f-54f2-53f3-20dd-22fc7dff55f9

  

[joergsesterhenn]

Hi @WeihanLi . I just ran into this error too. Did you find a solution to this? Where do I have to add true in my project. I wasn't able to find this properly explained anywhere. Or is this just an issue of incompatible vs/dotnet/msbuild versions? Or can I overwrite this on the comandline?

[WeihanLi]

Hi @joergsesterhenn , I just add <IsTestProject>true</IsTestProject> to my test project, maybe issue of the dotnet SDK, I'd like to reopen it to get more info from the team, your SDK/environment information may helpful for the team

[CarlosLanderas]

I have the same problem:

1>Done building target "ShowCallOfVSTestTaskWithParameter" in project "FunctionalTests.csproj".
Done executing task "CallTarget".

   **Task "Microsoft.TestPlatform.Build.Tasks.VSTestTask" skipped, due to false condition; ('$(IsTestProject)' == 'true') was evaluated as ('' == 'true').**

When using DotNetCoreCLI@2 command 'test' with SDK 5.0.101

Adding

<IsTestProject>true</IsTestProject>

solved the problem

[KalleOlaviNiemitalo]

@CarlosLanderas, does your project reference Microsoft.NET.Test.Sdk, and which version? Its Microsoft.NET.Test.Sdk.props would set the IsTestProject property.

[CarlosLanderas]

@CarlosLanderas, does your project reference Microsoft.NET.Test.Sdk, and which version? Its Microsoft.NET.Test.Sdk.props would set the IsTestProject property.

Yes, we are using Directory.Build.Targets with the following version:

<PackageReference Update="Microsoft.NET.Test.Sdk" Version="16.7.1" />

csproj references:

<ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" />
    <PackageReference Include="xunit" />
    <PackageReference Include="xunit.runner.visualstudio" />
    <PackageReference Include="coverlet.collector" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Design" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" />
    <PackageReference Include="Microsoft.Extensions.Configuration" />
    <PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Json" />
    <PackageReference Include="Moq" />
    <PackageReference Include="Respawn" />
    <PackageReference Include="FluentAssertions" />
</ItemGroup>

PD: I have another project with version 16.9.4 and I have the same problem

PD2: If I move the dotnet test execution to a shell script and trigger it in Azure Devops it works. Could it be some kind of problem with the Azure Devops Dotnet CLI task?

[oluatte]

Please fix this issue or make it clear somewhere in docs. I spent about five hours battling this (ADO Pipelines refusing to run tests) yesterday. No fun, no fun at all

[Boussalem]

Hello, we are having the same error here.

Can you give us feedbacks please on the issue.
Why should we add Property while we reference "Microsoft.NET.Test.Sdk"?

[ChristoWolf]

I am observing this issue in general (i.e. when executing dotnet test locally, or in Azure Pipelines using the DotNetCoreCLI@2task) when trying to run tests targeting an already built project file using the --no-build flag.
Adding the <IsTestProject>true</IsTestProject> (or the equivalent MSBuild property -p:IsTestProject=true) leads to the expected behavior, i.e. that the tests are actually run.

[edumserrano]

I'm also experiencing this when building my project in GitHub.
This is the workflow: https://github.com/edumserrano/github-issue-forms-parser/actions/workflows/build-test.yml.
You can see that in this run it failed because test NuGet packages defined in the Directory.build.props were not restored. These packages have a condition as follows: Condition="'$(IsTestProject)' == 'true'".

This was the error:

Run dotnet build GitHubIssueFormsParser/GitHubIssueFormsParser.sln -c Release -warnaserror --no-restore --no-incremental
Microsoft (R) Build Engine version 17.1.1+a02f73656 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

GitHubIssuesParserCli -> /home/runner/work/github-issue-forms-parser/github-issue-forms-parser/GitHubIssueFormsParser/src/GitHubIssuesParserCli/bin/Release/net6.0/GitHubIssuesParserCli.dll
Error: /home/runner/work/github-issue-forms-parser/github-issue-forms-parser/GitHubIssueFormsParser/tests/GitHubIssuesParserCli.Tests/GlobalUsings.cs(7,[14](https://github.com/edumserrano/github-issue-forms-parser/runs/6050059876?check_suite_focus=true#step:7:14)): error CS0[24](https://github.com/edumserrano/github-issue-forms-parser/runs/6050059876?check_suite_focus=true#step:7:24)6: The type or namespace name 'Shouldly' could not be found (are you missing a using directive or an assembly reference?) [/home/runner/work/github-issue-forms-parser/github-issue-forms-parser/GitHubIssueFormsParser/tests/GitHubIssuesParserCli.Tests/GitHubIssuesParserCli.Tests.csproj]

Build FAILED.

Error: /home/runner/work/github-issue-forms-parser/github-issue-forms-parser/GitHubIssueFormsParser/tests/GitHubIssuesParserCli.Tests/GlobalUsings.cs(7,14): error CS0246: The type or namespace name 'Shouldly' could not be found (are you missing a using directive or an assembly reference?) [/home/runner/work/github-issue-forms-parser/github-issue-forms-parser/GitHubIssueFormsParser/tests/GitHubIssuesParserCli.Tests/GitHubIssuesParserCli.Tests.csproj]
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:08.01
Error: Process completed with exit code 1.

After I added <IsTestProject>true</IsTestProject> to the test project then the build step worked as expected.

Some other notes:

• I can build and run tests fine locally via Visual Studio.
• I cannot build the solution locally using the dotnet CLI. I get the same error as shown above.
• The test project is referencing Microsoft.NET.Test.Sdk version 17.1.0.

Right now the workaround to add the IsTestProject property let's me build but that shouldn't be needed if I'm understanding what is being said in this issue.

Any ideas what might be wrong here? Let me know if you need more details.

[mjolka]

I ran into this problem when building the project as root and trying to run the tests as a non-root user.

I believe the problem comes down to Microsoft.NET.Test.Sdk.props being unreadable by the user trying to run the tests, as it's under /root.

A workaround is setting NUGET_PACKAGES to /tmp/.nuget/packages.

To see the issue:

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
RUN dotnet new xunit --name TestProject --output . --no-restore
ARG NUGET_PACKAGES
ENV NUGET_PACKAGES=${NUGET_PACKAGES}
RUN dotnet restore TestProject.csproj
RUN dotnet build TestProject.csproj --configuration Release --output /app/build --no-restore

Running the tests as root works:

$ docker build -t issue-3790 --quiet .
sha256:02b63fadacaf72fd215a105dcdf366d36c58eef08f0e9618a732cede18df7d6d
$ docker run --rm issue-3790 dotnet test --no-build --output /app/build
Test run for /app/build/TestProject.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.2.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: < 1 ms - /app/build/TestProject.dll (net6.0)
$ 

Running the tests as a non-root user gives no output:

$ docker run --rm --user 1000 -e DOTNET_CLI_HOME=/tmp issue-3790 dotnet test --no-build --output /app/build
$ 

But if you set NUGET_PACKAGES=/tmp/.nuget/packages, you get this:

$ docker build -t issue-3790 --quiet --build-arg NUGET_PACKAGES=/tmp/.nuget/packages . 
sha256:655c8e20298adcf7fa8f187c4c66f2189d1c5fab7e2aa356b09c60d8f86521c2
$ docker run --rm --user 1000 -e DOTNET_CLI_HOME=/tmp issue-3790 dotnet test --no-build --output /app/build
Test run for /app/build/TestProject.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.2.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: < 1 ms - /app/build/TestProject.dll (net6.0)
$ 

[andrei-epure-sonarsource]

I've hit this problem as well, with inconsistent behavior on Azure Pipelines CI. We have two jobs which run in parallel (one with .NET Framework 4.8 UTs, another with .NET 6 UTs) and it's very strange that one of them is failing when using coverlet with dotnet test. It's either the 4.8 or the 6 job failing while the other passes.

When examining the logs of a successful vs failed job (for the same TFM, different executions), we've noticed that the difference is the IsTestProject property which sometimes is not set for the UT project (the CSPROJ referencing <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" /> ). It's quite difficult to debug why this is happening.

Can there be something in the execution of dotnet test that overwrites that property, or misses picking it up from Microsoft.NET.Test.Sdk.props?

[giggio]

I have found the solution for this problem. This is my scenario:

1. I'm running in a CI machine
2. It does not have a cache for packages, so there is no ~/.nuget directory
3. I have all the binaries available, restored from an artifact

If I run this, it fails:

dotnet test --configuration Release --list-tests --nologo --no-build

If I dotnet restore, then run it, it works.

I with I did not have to restore again. Adding <IsTestProject>true</IsTestProject> works, too.

Not having the package at ~\.nuget\packages\microsoft.net.test.sdk\17.2.0\build\netcoreapp2.1\ is making it not find the .targets and .props files, I guess.

[edumserrano]

Given the last few comments I'd like to add that for my scenario, even if I run an admin shell on my Windows dev machine and whether or not I do a nuget restore before I try to build/run the tests I still get the same error due to the packages on the Directory.Build.props under the condition <ItemGroup Condition="'$(IsTestProject)' == 'true'"> not being restored at all.

The workaround for me remains to add the IsTestProject property to the csproj of my test projects.

[dominikjeske]

I have same issue as @edumserrano - we have condition in Directory.Build.props to include some nuget packages only to test projects. This solution worked very well but something changed and now it is not working. It looks like it only works in VS but not in CLI.

[OwnageIsMagic]

@edumserrano Directory.Build.props is almost the first thing evaluated during build (even before SDK). So items defined here https://github.com/edumserrano/github-issue-forms-parser/blob/6966f7809456736f6d87d27d8414a4bf96d5a68e/GitHubIssueFormsParser/Directory.Build.props#L39 are actually never evaluated.

[baronfel]

@OwnageIsMagic is correct - if you need a more convenient timing for these kinds of modifications, use a Directory.Build.targets file - there's a template for both targets and props files in the .Net 8 SDK you can use as a starter.

[edumserrano]

@OwnageIsMagic I'm not sure what you're trying to say with your comment. That condition you linked most definitely works for my setup in that repo...

[OwnageIsMagic]

@edumserrano Directory.Build.props evaluated before csproj. There is no way to affect conditions in Build.props from csproj. You probably depend on some Visual Studio quirk or passing IsTestProject=true from cli or env var (or some other msbuild extension point or those packages are not used or they flow transitively from another project).
If you are sure that you are not passing it directly, you should open bug on Visual Studio. Your setup SHOULD NOT work.

[edumserrano]

@OwnageIsMagic I don't mean to contradict you but if you read my comment here you can see that when running a dotnet test command on the GitHub pipeline things don't work without the IsTestProject set on the csproj.

They do work once that property is added.

Nothing else changes in the above setup, not even the Directory.Build.props. So either what you're saying isn't true for this case for whatever reason or indeed there's something else weird going on.

[OwnageIsMagic]

@edumserrano try cleaning up your nuget cache folder (dotnet nuget locals all --clear) and run git clean -xdf in project. You should see error on first build even with property set in csproj.
Anyway I strongly suggest move this block to Build.targets so it will be evaluated after property set in csproj. This will also allow Test.Sdk to set it accordingly, so you can actually drop it.

[edumserrano]

@OwnageIsMagic Do note that the problem is running on the GitHub pipeline which would have a fresh container on each run. No point in doing git clean -xdf for instance on the GitHub pipeline where this issue is also reproducible right?

Thanks for the feedback.

I do think this GitHub issue is still unresolved. The IsTestProject property on the tests csproj is not required when building via Visual Studio but it then is required when using dotnet CLI.

[cdschneider]

The team I work on was recently affected by the issue where tests are skipped with dotnet test (the same symptoms described earlier in this issue). The thread here helped in resolving the issue, where adding <IsTestProject>true</IsTestProject> at a project-level solved our problems.

However, I began looking deeper at the dotnet/test-templates project and wanted to ask if this issue and dotnet/test-templates#109 should both be closed. It appears that in the latest LTS versions of the test-project templates do already have this set at the project-level.

[lukebakken]

@giggio - thank you for your detailed description in your comment. It has allowed me to fix the same issue here - basically I am only uploading build artifacts between GitHub actions jobs, and not doing a build or NuGet restore.

[Evangelink]

Closing this issue as it's fixed on test-templates side.