Upgrade to version 3.0.0/3.1.0 causes FileNotFoundException (Newtonsoft assembly)

[mdrgeniebelt]

Hi,

At first I thought this is APM package that is causing this issue but after upgrading/downgrading different packages for a day I found out that it is coverlet.collector that is causing it. The issue only shows up in docker container though, we're using mcr.microsoft.com/dotnet/sdk:5.0-alpine image.

Exception 01 ===================================
Type: System.IO.FileNotFoundException
Source: Elastic.Apm.Report.Serialization.PayloadItemSerializer, Elastic.Apm, Version=1.7.1.0, Culture=neutral, PublicKeyToken=ae7400d2c189cf22
Message: Could not load file or assembly 'Newtonsoft.Json, Version=11.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. The system cannot find the file specified.
Trace:    at Elastic.Apm.Report.Serialization.PayloadItemSerializer..ctor(IConfigurationReader configurationReader)
   at Elastic.Apm.Report.PayloadSenderV2..ctor(IApmLogger logger, IConfigSnapshot config, Service service, System system, IApmServerInfo apmServerInfo, HttpMessageHandler httpMessageHandler, String dbgName, Boolean isEnabled)
   at Elastic.Apm.AgentComponents..ctor(IApmLogger logger, IConfigurationReader configurationReader, IPayloadSender payloadSender, IMetricsCollector metricsCollector, ICurrentExecutionSegmentsContainer currentExecutionSegmentsContainer, ICentralConfigFetcher centralConfigFetcher, IApmServerInfo apmServerInfo)
   at Elastic.Apm.AgentComponents..ctor(IApmLogger logger, IConfigurationReader configurationReader, IPayloadSender payloadSender)
   at Elastic.Apm.AspNetCore.ApmMiddlewareExtension.UseElasticApm(IApplicationBuilder builder, IConfiguration configuration, IDiagnosticsSubscriber[] subscribers)
   at Elastic.Apm.NetCoreAll.ApmMiddlewareExtension.UseAllElasticApm(IApplicationBuilder builder, IConfiguration configuration)
   at Organisation.Infrastructure.FrameworkExtensions.UseElasticApmMetricsConfigured(IApplicationBuilder app, IConfiguration configuration, IWebHostEnvironment webHostEnvironment) in /app/src/Organisation/Infrastructure/FrameworkExtensions.cs:line 93
   at Organisation.Startup.Configure(IApplicationBuilder app, ILogger`1 logger) in /app/src/Organisation/Startup.cs:line 108
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.<>c__DisplayClass4_0.<Build>b__0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass15_0.<UseStartup>b__1(IApplicationBuilder app)
   at Microsoft.AspNetCore.Mvc.Filters.MiddlewareFilterBuilderStartupFilter.<>c__DisplayClass0_0.<Configure>g__MiddlewareFilterBuilder|0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
Location: 
Method: Void .ctor(Elastic.Apm.Config.IConfigurationReader) (0, 0)

More details here:

elastic/apm-agent-dotnet#1117

[mdrgeniebelt]

Just to make it clear, downgrading coverlet.collector to 1.3.0 makes this exception disappear so it is not APMs fault (the exception message is misleading...).

[tdg5]

I am hitting this too! In my case it manifests like a YamlDotNet problem, but working through all the dependencies I found that it was coverlet.collector v3.0.0 that was forcing Newtonsoft.Json 9x into the build

aaubry/YamlDotNet#559

[MarcoRossignoli]

Are you all using collectors?

[mdrgeniebelt]

Are you all using collectors?

Sorry maybe a stupid question but what do you mean by that?

I have two nuget packages:

• coverlet.collector v1.3.0
• coverlet.msbuild v3.0.0

[MarcoRossignoli]

can you write your command line?

[mdrgeniebelt]

So we're using github actions to run unit tests and generate coverage report, this is the part of yaml for pull request checks:

      - name: Run unit tests with dotnet
        run:  dotnet test -p:CollectCoverage=true -p:CoverletOutput=../TestResults/ -p:CoverletOutputFormat=lcov -p:Include=\"[*]*.Features.*\" -p:ExcludeByFile=\"**/Database/*,**/Infrastructure/*\"

      - name: Comment the pull request with Code Coverage
        uses: romeovs/lcov-reporter-action@v0.2.11
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          lcov-file: ./test/TestResults/coverage.info

Apart from that we're not executing anything related to code coverage inside the docker.

[MarcoRossignoli]

ok you're using msbuild integration.
Can you try to test with collectors please? https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md

Only to understand if it's an issue related to msbuild integration or not.

[billybraga]

I believe the issue is coverlet.collector nuget package doesn't specify dependencies, but still includes Newtonsoft.Json.dll.

From the detailed output publish, I see this :

Copying file from "/home/--ME--/.nuget/packages/coverlet.collector/3.0.1/build/netstandard1.0/Newtonsoft.Json.dll" to "/home/local/--ME--/dev/projects/Xxx/tests/Xxx.Tests.Lib/bin/Release/net5.0/publish/Newtonsoft.Json.dll".

[MarcoRossignoli]

@billybraga if you don't publish and run dotnet test on csproj do you see that line?Also the collectors deps is added as private assets? https://docs.microsoft.com/en-us/nuget/consume-packages/package-references-in-project-files#controlling-dependency-assets

<PackageReference Include="coverlet.collector" Version="3.0.1">
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>

[billybraga]

I don't have the problem with dotnet test (the dll is not copied from coverlet, as expected).

Here is my package reference:

      <PackageReference Include="coverlet.collector" Version="3.0.1">
        <PrivateAssets>all</PrivateAssets>
        <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      </PackageReference>

(it's what dotnet added automatically after installing coverlet.collector)

[MarcoRossignoli]

#1045 (comment)

@mdrgeniebelt you're using msbuild integration, can you remove the package reference to coverlet.collector and retry?

[mdrgeniebelt]

@MarcoRossignoli looks like removing reference to coverlet.collector fixes this :) Thank you! (I updated msbuild package to 3.0.2 also).

[MattAfflek]

Why the issue is closed?
Despite the fact that removing reference to this package (stopping using it) also helped me, I do not think it is solution,

I have the same issue.

Steps to reproduce:

1. Create new console project .net core 3.1 (will probably work with 5.0 as well)
2. Add Newtonsoft.Json version 13.0.1 (I bet any version >= 11.0.0 will work)
3. Add the next line in Program.cs
   Console.WriteLine(JsonConvert.SerializeObject(new string[] { "Small", "Medium", "Large" }));
   Or use some features added after version 9.0.1
   Console.WriteLine(JsonConvert.SerializeObject(new string[] { "Small", "Medium", "Large" }, new JsonSerializerSettings { Converters = { new StringEnumConverter(new KebabCaseNamingStrategy()) } }));
4. Add new xunit project (check that it includes "coverlet.collector")
5. Use dotnet publish to publish code. (E.g. dotnet publish -r win-x64 -c Release --self-contained true /p:useapphost=true)
6. Check that in folder where it is published Newtonsoft.Json has version 9.0.1
7. Run

[MattAfflek]

@billybraga is absolutely right about it.
coverlet.collector nuget package doesn't specify dependencies, but still includes Newtonsoft.Json.dll.

[JPThorne]

This is also happening for us, not sure why issue is closed, we can't just "not use the package" as a solution?

[billybraga]

Still happening

[MarcoRossignoli]

Reopening because the issue is still present with collector.

[MarcoRossignoli]

Can someone provide a small repro(with collector usage)?

[billybraga]

https://github.com/billybraga/CoverletNewtonsoftRepro/tree/main

[MarcoRossignoli]

@billybraga I saw that you're publishing a project. I expect that you don't reference coverlet collector for a project like all/exe/etc...it should be referenced only for test project and used with dotnet test.... command. Are you referencing the collectors from a non test project?

[billybraga]

I updated the repo to remove the coverlet collector from the console project and add a test project with the coverlet collector. The same error occurs when running the published console app.

[MarcoRossignoli]

@billybraga you should avoid -o publish because running this from repo root will publish all project to that directory and so you're moving test and non test project to same output folder.
If you run only dotnet publish it will create two different publish folder and non test one won't contain old newtosoft lib.

Anyway the issue will be present if you publish test project. Does it work if you simply run dotnet test --collect:"XPlat Code Coverage" (cleanup workspace before to be sure to have clean bin output folder, something like git clean -fdx)

[billybraga]

I solved my problem by adding IsPublishable=false in csproj of test projects (instead of removing the test dlls in CI/CD before building final docker image with dlls).

As for running dotnet test ..., this has never been a problem.

I still think coverlet shouldn't add a Newtonsoft dll directly instead of relying on nuget dependencies, but personally I'm not having problems anymore with that since using IsPublishable=false like stated above.

[daveMueller]

I'm working on this issue. Think I already have a hint.

[MarcoRossignoli]

Hi all!

Can someone dogfood this preview version with the fix attempt and tell me if it works?
https://f.feedz.io/marcorossignoli/coverletunofficial/nuget/index.json version 3.1.1-preview.10

[aleto-ms]

Can someone dogfood this preview version with the fix attempt and tell me if it works? https://f.feedz.io/marcorossignoli/coverletunofficial/nuget/index.json version 3.1.1-preview.10

I can confirm, the publish now doesn't override Newtonsoft.Json.

P.S.
After spending so much time trying code coverage work, need for explicit publish which in turned resulted in this issue.. I share a lot @billybraga's view

I still think coverlet shouldn't add a Newtonsoft dll directly instead of relying on nuget dependencies,

I obviously don't know the original reasons to use build instead of 'lib', but the latter would solve the need of explicit publish requirement and enable leveraging native Nuget dependencies flow.