The dotnet store is incorrectly including BCL assemblies when targeting .NET Core 3.1 #10973

normj · 2020-03-24T07:34:26Z

Using the following store manifest that only includes Newtonsoft.Json

<Project Sdk="Microsoft.NET.Sdk">
  <ItemGroup>
    <PackageReference Include="Newtonsoft.Json" Version="9.0.1" />
  </ItemGroup>
</Project>

I execute the following command to create a store for .NET Core 2.1:

dotnet store --manifest .\manifest.xml -f netcoreapp2.1 -o ./store2.1 -r linux-x64 --skip-optimization

the store artifact.xml contains only Newtonsoft.Json as expected.

<StoreArtifacts>
  <Package Id="Newtonsoft.Json" Version="9.0.1" />
</StoreArtifacts>

But then creating the store for .NET Core 3.1 using the command:

dotnet store --manifest .\manifest.xml -f netcoreapp3.1 -o ./store3.1 -r linux-x64 --skip-optimization

Creates a store with a lot of runtime and system dlls which bloat the package store and also those assembles fail to the store optimization.

<StoreArtifacts>
  <Package Id="Microsoft.CSharp" Version="4.0.1" />
  <Package Id="Newtonsoft.Json" Version="9.0.1" />
  <Package Id="runtime.any.System.Collections" Version="4.0.11" />
  <Package Id="runtime.any.System.Diagnostics.Tools" Version="4.0.1" />
  <Package Id="runtime.any.System.Globalization" Version="4.0.11" />
  <Package Id="runtime.any.System.IO" Version="4.1.0" />
  <Package Id="runtime.any.System.Reflection" Version="4.1.0" />
  <Package Id="runtime.any.System.Reflection.Extensions" Version="4.0.1" />
  <Package Id="runtime.any.System.Reflection.Primitives" Version="4.0.1" />
  <Package Id="runtime.any.System.Resources.ResourceManager" Version="4.0.1" />
  <Package Id="runtime.any.System.Runtime" Version="4.1.0" />
  <Package Id="runtime.any.System.Runtime.Handles" Version="4.0.1" />
  <Package Id="runtime.any.System.Runtime.InteropServices" Version="4.1.0" />
  <Package Id="runtime.any.System.Text.Encoding" Version="4.0.11" />
  <Package Id="runtime.any.System.Text.Encoding.Extensions" Version="4.0.11" />
  <Package Id="runtime.any.System.Threading.Tasks" Version="4.0.11" />
  <Package Id="runtime.unix.System.Diagnostics.Debug" Version="4.0.11" />
  <Package Id="runtime.unix.System.IO.FileSystem" Version="4.0.1" />
  <Package Id="runtime.unix.System.Private.Uri" Version="4.0.1" />
  <Package Id="runtime.unix.System.Runtime.Extensions" Version="4.1.0" />
  <Package Id="System.Dynamic.Runtime" Version="4.0.11" />
  <Package Id="System.IO.FileSystem.Primitives" Version="4.0.1" />
  <Package Id="System.Linq" Version="4.1.0" />
  <Package Id="System.Linq.Expressions" Version="4.1.0" />
  <Package Id="System.ObjectModel" Version="4.0.12" />
  <Package Id="System.Reflection.Emit" Version="4.0.1" />
  <Package Id="System.Reflection.Emit.ILGeneration" Version="4.0.1" />
  <Package Id="System.Reflection.Emit.Lightweight" Version="4.0.1" />
  <Package Id="System.Reflection.TypeExtensions" Version="4.1.0" />
  <Package Id="System.Runtime.Serialization.Primitives" Version="4.1.1" />
  <Package Id="System.Text.RegularExpressions" Version="4.1.0" />
  <Package Id="System.Threading" Version="4.0.11" />
  <Package Id="System.Threading.Tasks.Extensions" Version="4.0.0" />
  <Package Id="System.Xml.ReaderWriter" Version="4.0.11" />
  <Package Id="System.Xml.XDocument" Version="4.0.11" />
</StoreArtifacts>

This change in behavior seems to make the dotnet store command a broken experience.

@nguerrera We have talked about dotnet store in the past. Can you shed any light on why this is happening now?

The text was updated successfully, but these errors were encountered:

normj · 2020-03-31T03:34:10Z

Friendly ping on this to get some attention. This is breaking some of our scenarios for .NET Core 3.1.

rpopovych · 2020-04-03T14:40:54Z

This is a bummer. Prevents us from using 3.1 in some of our AWS Lambdas. Hoping there is an easy resolution/workaround for this issue.

davidruhmann · 2020-04-10T00:04:02Z

Wonder if the root cause is related to this issue dotnet/runtime#2836;

Found the issue referenced in the code to handle an additional file being stored. https://github.com/dotnet/sdk/blob/master/src/Tests/Microsoft.NET.Publish.Tests/GivenThatWeWantToStoreAProjectWithDependencies.cs#L92

From a quick look at different versions, this only seems to happen for Newtonsoft.Json versions < 11. Only major change that I initially see to cause this is that version 11 starts referencing netstandard2.0.

dsplaisted · 2020-04-14T19:51:42Z

This is a result of changes we made in .NET Core 3. We no longer use PackageReferences to represent .NET Core Framework. This is discussed here: dotnet/designs#50

The old version of NewtonSoft.Json has dependencies to packages which are part of .NET Core. Because we don't represent .NET Core as a PackageReference anymore, the dotnet store logic no longer knows to exclude them from the store.

The ideal fix to this would likely be NuGet/Home#7344, and possibly additional changes in dotnet store to use it.

Possible workarounds:

Use a later version of NewtonSoft.Json (or in general, use packages that don't have these older dependencies, which will generally be the case if the package was built for .NET Core 2.0 or .NET Standard 2.0 or higher).
Exclude each unneeded package from the manifest by adding the following

  <Target Name="AddPackagesToPrune" AfterTargets="PrepforRestoreForComposeStore">
    <ItemGroup>
      <PackagesToPrune Include="Microsoft.CSharp" />
      <PackagesToPrune Include="System.Collections" />
      <PackagesToPrune Include="System.Diagnostics.Debug" />
      <PackagesToPrune Include="System.Dynamic.Runtime" />
      <PackagesToPrune Include="System.Globalization" />
      <PackagesToPrune Include="System.IO" />
      <PackagesToPrune Include="System.Linq" />
      <PackagesToPrune Include="System.Linq.Expressions" />
      <PackagesToPrune Include="System.ObjectModel" />
      <PackagesToPrune Include="System.Reflection" />
      <PackagesToPrune Include="System.Reflection.Extensions" />
      <PackagesToPrune Include="System.Resources.ResourceManager" />
      <PackagesToPrune Include="System.Runtime" />
      <PackagesToPrune Include="System.Runtime.Extensions" />
      <PackagesToPrune Include="System.Runtime.Serialization.Primitives" />
      <PackagesToPrune Include="System.Text.Encoding" />
      <PackagesToPrune Include="System.Text.Encoding.Extensions" />
      <PackagesToPrune Include="System.Text.RegularExpressions" />
      <PackagesToPrune Include="System.Threading" />
      <PackagesToPrune Include="System.Threading.Tasks" />
      <PackagesToPrune Include="System.Xml.ReaderWriter" />
      <PackagesToPrune Include="System.Xml.XDocument" />
    </ItemGroup>
    <PropertyGroup>
      <PackagesToPrune>$(PackagesToPrune);@(PackagesToPrune)</PackagesToPrune>
    </PropertyGroup>
  </Target>

Note that if there is a PackageToPrune which is not part of the dependency graph, you will get a NullReferenceException from one of the tasks.

normj · 2020-04-15T04:42:44Z

Interesting. In my case dotnet store is used as a general tool and I don't have control what packages are in the manifest. I happen to pick Newtonsoft.Json as an example because one of my libraries has a dependency on that version that is commonly put into the store along with the customers packages.

I have the option to make a modifications to a copy of the inputted manifest and use that when calling dotnet store. I need to come up with a way given a manifest be able to compute what packages to prune or at a minimum detect that package will bring in framework packages and they should upgrade their package versions.

dsplaisted · 2020-04-21T23:59:38Z

@normj Could you update your library which depends on NewtonSoft.Json to depend on at least 11.0.1? That should help mitigate the issue.

normj · 2020-04-22T06:16:39Z

I can, but I can't control whether somebody uses our tool that relies dotnet store with dependencies that will also have this problem. Is there anyway for me to programmatically detect when a store manifest would potentially bring in these bad refs? I'm afraid I would need to build the whole dependency graph myself first and inspect before calling dotnet store.

davidruhmann · 2020-04-22T21:19:18Z

@dsplaisted, do you see any issues with this type of workaround until NuGet/Home#7344 is resolved?

Run dotnet store on the initial manifest.
Create a new manifest file with a prune list (consisting of entries of dependencies in the artifact.xml that also exist in the runtime folder (e.g. /usr/local/share/dotnet/shared/Microsoft.NETCore.App/3.1.3)
- from System.Runtime.InteropServices.RuntimeEnvironment.GetRuntimeDirectory() but for the target framework.
Delete the old output.
Then run dotnet store again on the new manifest file.

Am I missing any major aspect regarding this?

dsplaisted · 2020-04-22T22:05:45Z

@davidruhmann I think something like that could work, but mapping from the dependencies in the runtime folder to packages that should be excluded may not be simple. You have to consider not just the file name but the version number, and also map it to the right NuGet package.

davidruhmann · 2020-04-22T22:24:28Z

Appreciate the quick reply!

Version did cross my mind, but wouldn't that just be for higher versioned references? Which, if I understand correctly, could still be resolved by the store consumer? Meaning the store would possibly be missing some elements, but not the flip side of including unnecessary elements.

As for the naming of nuget packages to BCL components, are there many differences?

I am thinking of this workaround in a scope of 80% success or something like that. If it can solve the majority of conflicts for the current time until NuGet/Home#7344 is complete; that should be good enough vs making the workaround too complex. Even if the other 20% result in failure, the original bloated store could still be used or fail the process entirely saying unsupported.

clearwaterstream · 2020-04-22T22:41:01Z

@davidruhmann I must admit I have not patience for gardening, but pruning I did try. Initial manifest -> dotnet store -> look at resulting artifact.xml -> add the extraneous packages to prune list on initial manifest -> dotnet store again.

What I found is that the AddPackagesToPrune would choke on certain packages. Let's do a simple example.

Initial manifest:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <ItemGroup>
	<PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
  </ItemGroup>
</Project>

dotnet store --manifest initial.xml -f netcoreapp3.1 -o bin/dotnetcore/store -r linux-x64 --skip-optimization

Output is:

<StoreArtifacts>
  <Package Id="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
  <Package Id="Microsoft.CSharp" Version="4.3.0" />
  <Package Id="Microsoft.IdentityModel.JsonWebTokens" Version="5.5.0" />
  <Package Id="Microsoft.IdentityModel.Logging" Version="5.5.0" />
  <Package Id="Microsoft.IdentityModel.Protocols" Version="5.5.0" />
  <Package Id="Microsoft.IdentityModel.Protocols.OpenIdConnect" Version="5.5.0" />
  <Package Id="Microsoft.IdentityModel.Tokens" Version="5.5.0" />
  <Package Id="Newtonsoft.Json" Version="10.0.1" />
  <Package Id="runtime.any.System.Collections" Version="4.3.0" />
  <Package Id="runtime.any.System.Diagnostics.Tools" Version="4.3.0" />
  <Package Id="runtime.any.System.Diagnostics.Tracing" Version="4.3.0" />
  <Package Id="runtime.any.System.Globalization" Version="4.3.0" />
  <Package Id="runtime.any.System.IO" Version="4.3.0" />
  <Package Id="runtime.any.System.Reflection" Version="4.3.0" />
  <Package Id="runtime.any.System.Reflection.Extensions" Version="4.3.0" />
  <Package Id="runtime.any.System.Reflection.Primitives" Version="4.3.0" />
  <Package Id="runtime.any.System.Resources.ResourceManager" Version="4.3.0" />
  <Package Id="runtime.any.System.Runtime" Version="4.3.0" />
  <Package Id="runtime.any.System.Runtime.Handles" Version="4.3.0" />
  <Package Id="runtime.any.System.Runtime.InteropServices" Version="4.3.0" />
  <Package Id="runtime.any.System.Text.Encoding" Version="4.3.0" />
  <Package Id="runtime.any.System.Text.Encoding.Extensions" Version="4.3.0" />
  <Package Id="runtime.any.System.Threading.Tasks" Version="4.3.0" />
  <Package Id="runtime.unix.System.Diagnostics.Debug" Version="4.3.0" />
  <Package Id="runtime.unix.System.IO.FileSystem" Version="4.3.0" />
  <Package Id="runtime.unix.System.Private.Uri" Version="4.3.0" />
  <Package Id="runtime.unix.System.Runtime.Extensions" Version="4.3.0" />
  <Package Id="System.Buffers" Version="4.3.0" />
  <Package Id="System.Collections.NonGeneric" Version="4.3.0" />
  <Package Id="System.Collections.Specialized" Version="4.3.0" />
  <Package Id="System.ComponentModel" Version="4.3.0" />
  <Package Id="System.ComponentModel.Primitives" Version="4.3.0" />
  <Package Id="System.ComponentModel.TypeConverter" Version="4.3.0" />
  <Package Id="System.Dynamic.Runtime" Version="4.3.0" />
  <Package Id="System.Globalization.Extensions" Version="4.3.0" />
  <Package Id="System.IdentityModel.Tokens.Jwt" Version="5.5.0" />
  <Package Id="System.IO.FileSystem.Primitives" Version="4.3.0" />
  <Package Id="System.Linq" Version="4.3.0" />
  <Package Id="System.Linq.Expressions" Version="4.3.0" />
  <Package Id="System.ObjectModel" Version="4.3.0" />
  <Package Id="System.Reflection.Emit" Version="4.3.0" />
  <Package Id="System.Reflection.Emit.ILGeneration" Version="4.3.0" />
  <Package Id="System.Reflection.Emit.Lightweight" Version="4.3.0" />
  <Package Id="System.Reflection.TypeExtensions" Version="4.3.0" />
  <Package Id="System.Runtime.Numerics" Version="4.3.0" />
  <Package Id="System.Runtime.Serialization.Formatters" Version="4.3.0" />
  <Package Id="System.Runtime.Serialization.Primitives" Version="4.3.0" />
  <Package Id="System.Security.Cryptography.Cng" Version="4.5.0" />
  <Package Id="System.Text.RegularExpressions" Version="4.3.0" />
  <Package Id="System.Threading" Version="4.3.0" />
  <Package Id="System.Threading.Tasks.Extensions" Version="4.3.0" />
  <Package Id="System.Xml.ReaderWriter" Version="4.3.0" />
  <Package Id="System.Xml.XDocument" Version="4.3.0" />
  <Package Id="System.Xml.XmlDocument" Version="4.3.0" />
</StoreArtifacts>

initial.xml with one package added to prune:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <ItemGroup>
	<PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
  </ItemGroup>
  <Target Name="AddPackagesToPrune" AfterTargets="PrepforRestoreForComposeStore">
    <ItemGroup>
      <PackagesToPrune Include="Microsoft.CSharp" />
    </ItemGroup>
    <PropertyGroup>
      <PackagesToPrune>$(PackagesToPrune);@(PackagesToPrune)</PackagesToPrune>
    </PropertyGroup>
  </Target>
</Project>

Prune worked -- there are less packages in artifact.xml. Great.

Now, if I do this:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <ItemGroup>
	<PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
	<PackageReference Include="Amazon.Lambda.AspNetCoreServer" Version="5.0.0" />
  </ItemGroup>
  <Target Name="AddPackagesToPrune" AfterTargets="PrepforRestoreForComposeStore">
    <ItemGroup>
      <PackagesToPrune Include="Microsoft.CSharp" />
    </ItemGroup>
    <PropertyGroup>
      <PackagesToPrune>$(PackagesToPrune);@(PackagesToPrune)</PackagesToPrune>
    </PropertyGroup>
  </Target>
</Project>

I will get Microsoft.NET.ComposeStore.targets(420,5): error MSB4018: The "FilterResolvedFiles" task failed unexpectedly. error.

davidruhmann · 2020-04-22T23:13:54Z

@clearwaterstream Thanks for creating the proof of concept. I see, this error is because the Filter target is being run per dependency and the Amazon dependency is not pulling in Microsoft.CSharp. Have to step away, but wonder if we can inject in before the Filter target to dynamically build the prune list per dependency. Will take a look later but it will most likely hinge on how the FilterResolvedFiles target is written and if ResolvedFiles is accessible to a before target.
https://github.com/dotnet/sdk/blob/master/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.ComposeStore.targets#L420

dsplaisted · 2020-04-22T23:16:08Z

We can probably fix the NullReferenceException in the 5.0 SDK if that would be a significant help. IE then it would be OK to specify packages to prune that aren't in the package graph.

clearwaterstream · 2020-04-23T01:28:18Z

@davidruhmann you mentioned "... dynamically build the prune list per dependency."

and @dsplaisted you mentioned "... specify packages to prune that aren't in the package graph".

So would the prune list calculation logic know how to not add packages to the prune list that are legitimate dependencies? For example:

<PackageReference Include="MySql.Data" Version="8.0.19" />

Would pull in (to name a few):

<Package Id="SSH.NET" Version="2016.1.0" />
<Package Id="SshNet.Security.Cryptography" Version="1.2.0" />
<Package Id="BouncyCastle.NetCore" Version="1.8.3" />
<Package Id="Google.Protobuf" Version="3.6.1" />

Currently if I blindly take all the "extra" packages that are in the resultant artifact.xml and add them to the prune list then I would be over-pruning in some cases.

davidruhmann · 2020-04-23T03:43:30Z

@dsplaisted Yes, that would be nice.

@clearwaterstream To clarify, the list of excluded libraries for the use case is limited to what is found in the framework runtime (the BCLs). Sub-depedencies would not be part of the exclusion list unless manually specified.

Here is a working example excluding Microsoft.CSharp from artifact.xml; however the file still exists in the store. Instead of passing in PackagesToPrune, it is removing packages from the output ResolvedPackagesPublished.

<Project Sdk="Microsoft.NET.Sdk.Web">

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
    <PackageReference Include="Amazon.Lambda.AspNetCoreServer" Version="5.0.0" />
  </ItemGroup>

  <Target Name="PruneBCLs" AfterTargets="ComputeFilesToStore">
    <ItemGroup>
      <BCLsToPrune Include="Microsoft.CSharp" />
    </ItemGroup>
    <ItemGroup>
      <ResolvedPackagesPublished Remove="@(BCLsToPrune)" />
    </ItemGroup>
  </Target>

</Project>

Now, in the case that @normj originally posted above; the BCLsToPrune list could be populated with the filenames found in the runtime directory. (e.g. /usr/local/share/dotnet/shared/Microsoft.NETCore.App/3.1.3)

normj · 2020-04-23T05:13:15Z

@dsplaisted Is there no way the NullReferenceException could be ported to the 3.1 SDK? We generally stick with LTS versions which means we wouldn't get a fix till .NET 6.

dsplaisted · 2020-05-11T16:25:51Z

@normj The NullReferenceException should be fixed in the 3.1.400 SDK release.

normj · 2020-05-11T16:55:14Z

@dsplaisted Great looking forward to it!

niklr · 2020-05-23T11:08:09Z

I was in the process to give AWS Lambda layers with .NET Core a try and got stuck with the publish-layer command referencing this issue here. What are your current recommmendations for such a situation @normj ? Should we wait for 3.1.400 SDK to be released or should we try with TargetFramework=netcoreapp2.1 ? The latter seems not to work with the following dependencies:

Amazon.Lambda.APIGatewayEvents (v2.1.0)
Amazon.Lambda.Core (v1.1.0)
Amazon.Lambda.Serialization.SystemTextJs (v2.0.1)

normj · 2020-05-26T17:29:37Z

@niklr For the Lambda side the ball is in my court now to update the Lambda tooling to use the work around added in 3.1.400. I hope to get to that soon just juggling a few things right now. I think the issue is still worth leaving open as I think the current fix in 3.1.400 is a work around that requires users of the dotnet store command to do extra work and hopefully there will be a fix someday that doesn't require any extra work.

vkkpnm · 2020-06-24T19:24:16Z

Looking forward for this change. Any tentative timelines?

RNavaneethKumar · 2020-06-29T15:38:06Z

@normj - Glad to see your contributions to the corresponding repositories this month. Can we expect anything soon?

ppittle · 2020-07-17T23:34:57Z

@dsplaisted - any news on when 3.1.400 will be out? Looks like the team released 3.1.302 earlier this week on July 14th 2020.

Verified version 3.1.302 still has the Null Reference Exception bug on @clearwaterstream's sample:

manifest.xml:

<Project Sdk="Microsoft.NET.Sdk">
  <ItemGroup>
    <PackageReference Include="Newtonsoft.Json" Version="9.0.1" />
	<PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
	<PackageReference Include="Amazon.Lambda.AspNetCoreServer" Version="5.0.0" />
  </ItemGroup>
  <Target Name="AddPackagesToPrune" AfterTargets="PrepforRestoreForComposeStore">
    <ItemGroup>
      <PackagesToPrune Include="Microsoft.CSharp" />
    </ItemGroup>
    <PropertyGroup>
      <PackagesToPrune>$(PackagesToPrune);@(PackagesToPrune)</PackagesToPrune>
    </PropertyGroup>
  </Target>  
</Project>

console:

> dotnet --version
3.1.302
> dotnet store --manifest initial.xml -f netcoreapp3.1 -o bin/dotnetcore/store -r linux-x64 --skip-optimization

C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018: The "FilterResolvedFiles" task failed unexpectedly. [C:\projects\sandbox\dotnetStoreBug\manifest.xml]
C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018: System.NullReferenceException: Object reference not set to an instance of an object. [C:\projects\sandbox\dotnetStoreBug\manifest.xml]
C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018:    at Microsoft.NET.Build.Tasks.LockFileExtensions.GetTransitivePackagesList(LockFileTarget lockFileTarget, LockFileTargetLibrary package, IDictionary`2 libraryLookup) [C:\projects\sandbox\dotnetStoreBug\manifest.xml]
C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018:    at Microsoft.NET.Build.Tasks.ProjectContext.GetTransitiveList(String package) [C:\projects\sandbox\dotnetStoreBug\manifest.xml]
C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018:    at Microsoft.NET.Build.Tasks.FilterResolvedFiles.ExecuteCore() [C:\projects\sandbox\dotnetStoreBug\manifest.xml]
C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute() [C:\projects\sandbox\dotnetStoreBug\manifest.xml]
C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [C:\projects\sandbox\dotnetStoreBug\manifest.xml]
C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [C:\projects\sandbox\dotnetStoreBug\manifest.xml]

dsplaisted · 2020-07-17T23:47:02Z

@ppittle 3.1.400 will release together with Visual Studio 2019 version 16.7. I don't think we've announced when that will be, but you can see the previews we've released so far here: https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes-preview

clearwaterstream · 2020-07-18T00:07:05Z

@dsplaisted I've skimmed through the notes https://docs.microsoft.com/en-us/visualstudio/releasenotes/vs2019-mac-preview-relnotes and 3.1.400 is not mentioned there either. Do you happen to know if this may potentially make it into the 8.7 release of VS 4 Mac?

dsplaisted · 2020-07-18T00:17:18Z

@leecow Do you know if VS for Mac 8.7 will include .NET Core SDK 3.1.400? Thanks.

leecow · 2020-07-21T16:04:05Z

I believe that's correct. @mrward to confirm.

mrward · 2020-07-21T18:02:52Z

VS Mac 8.7 currently includes .NET 3.1.302.

I expect VS Mac 8.7 will at some point include .NET 3.1.400 since Visual Studio 2019 16.7 will be doing so.

bjorg · 2020-07-21T18:38:48Z

As far as I can tell, 3.1.400 is not released. If it is, I don't see it here: https://dotnet.microsoft.com/download/dotnet-core/3.1

bonjo · 2020-07-29T03:59:44Z

@ppittle 3.1.400 will release together with Visual Studio 2019 version 16.7. I don't think we've announced when that will be, but you can see the previews we've released so far here: https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes-preview

Hi @dsplaisted / @normj
Today I installed .NET Core 3.1.400-preview-015216 from here https://github.com/dotnet/installer#installers-and-binaries and I still getting the below message when I run the following command: dotnet lambda publish-layer CP-Layer --layer-type runtime-package-store --s3-bucket cp-site
Publishing runtime package store layers targeting .NET Core 3.1 is currently disabled due to an issue with the dotnet cli's 'store' command used to create the runtime package store.
Do you known something about when this issue will be solved?

dsplaisted · 2020-07-29T15:29:37Z

@bonjo It sounds like that message is coming from the dotnet lambda tooling. IE it hasn't been updated to understand that the issue in the .NET SDK has been fixed.

normj · 2020-07-29T16:43:04Z

@bonjo as @dsplaisted mentioned we (AWS) need to update the dotnet lambda tooling once 3.1.400 is released. Once the new version is released the work for updating the dotnet lambda tooling will be tracked here. aws/aws-extensions-for-dotnet-cli#113

bonjo · 2020-07-29T20:01:09Z

@normj I assume then that dotnet lambda tool doesn't have a preview version that works with .NET core 3.1.400 preview.
It's correct?

bjorg · 2020-07-29T20:38:53Z

@bonjo You're in luck! It's open source as well, so you can build it locally with the required modification to unblock yourself.
https://github.com/aws/aws-extensions-for-dotnet-cli

normj · 2020-07-29T21:12:30Z

@bonjo correct we don't have a preview version as we haven't started the work yet.

slavanap · 2020-08-09T12:51:49Z

Whoever it may concern,
Please pay attention that with recent VS update MSI package for 3.1.400 is installed which is undeletable. Might be a bug with installation/uninstallation procedure. I don't know where to send this info.

slavanap · 2020-08-09T12:56:42Z

I would appreciate if someone provides official MSI for 3.1.400 (x64). The message below appears after MSVS uninstall.

dsplaisted · 2020-08-12T19:14:17Z

@slavanap We've released a new patch. You can download 3.1.401 here: https://dotnet.microsoft.com/download/dotnet-core/3.1

devssharma · 2020-08-19T12:43:30Z

I am getting this message while publishing the layer into AWS lambda layers using .NET CORE 3.1 Version.

Could you please help me how can i publish the layers into aws using .net core 3.1. ?? Do we have any workaround to do this ??

bjorg · 2020-08-19T15:39:05Z

@devssharma You need to post that over here: aws/aws-extensions-for-dotnet-cli#113
This issue was the underlying cause, but AWS needs to update their tool to re-enable it.

normj · 2020-10-11T03:52:58Z

I have a PR out using the pruning workaround into the AWS Lambda tooling. aws/aws-extensions-for-dotnet-cli#144

@dsplaisted do you think there will ever be a better solution in the dotnet store command other then adding this pruning list? I hope so but if not then we can close this issue and people concern with the AWS use case should track the fix via the PR.

dsplaisted · 2020-10-11T19:58:54Z

Unfortunately I think it's unlikely that we'll fix this with a better solution. It would require NuGet/Home#7344, which I would like to see and would fix some other issues. However, it would be a major feature for NuGet, and probably only helps for cases where a .NET Core 1.x / .NET Standard 1.x package is referenced, which should happen less and less as time goes on. So it's probably not going to be worth prioritizing.

normj · 2020-10-12T00:28:21Z

Alright thanks for the pruning fix. That seems to work for our use case.

This was referenced Mar 24, 2020

.NET Core Lambda Layers for .NET Core 3.1 Broken aws/aws-extensions-for-dotnet-cli#113

Closed

Disable creation of .NET Core 3.1 Lambda layers due to a issue in the… aws/aws-extensions-for-dotnet-cli#114

Merged

marcpopMSFT assigned dsplaisted Apr 3, 2020

dsplaisted mentioned this issue Apr 24, 2020

Fix NullReferenceException in dotnet store #11422

Merged

normj mentioned this issue Oct 11, 2020

Add support for creating layers with .NET Core 3.1 aws/aws-extensions-for-dotnet-cli#144

Merged

normj closed this as completed Oct 12, 2020

dsplaisted mentioned this issue Oct 6, 2021

Error creating optimised runtime package store #21551

Closed

The dotnet store is incorrectly including BCL assemblies when targeting .NET Core 3.1 #10973

The dotnet store is incorrectly including BCL assemblies when targeting .NET Core 3.1 #10973

Comments

normj commented Mar 24, 2020 • edited Loading

normj commented Mar 31, 2020

rpopovych commented Apr 3, 2020 • edited Loading

davidruhmann commented Apr 10, 2020 • edited Loading

dsplaisted commented Apr 14, 2020

normj commented Apr 15, 2020

dsplaisted commented Apr 21, 2020

normj commented Apr 22, 2020

davidruhmann commented Apr 22, 2020

dsplaisted commented Apr 22, 2020

davidruhmann commented Apr 22, 2020

clearwaterstream commented Apr 22, 2020

davidruhmann commented Apr 22, 2020

dsplaisted commented Apr 22, 2020

clearwaterstream commented Apr 23, 2020

davidruhmann commented Apr 23, 2020 • edited Loading

normj commented Apr 23, 2020

dsplaisted commented May 11, 2020

normj commented May 11, 2020

niklr commented May 23, 2020

normj commented May 26, 2020

vkkpnm commented Jun 24, 2020

RNavaneethKumar commented Jun 29, 2020

ppittle commented Jul 17, 2020

dsplaisted commented Jul 17, 2020

clearwaterstream commented Jul 18, 2020

dsplaisted commented Jul 18, 2020

leecow commented Jul 21, 2020

mrward commented Jul 21, 2020

bjorg commented Jul 21, 2020

bonjo commented Jul 29, 2020 • edited Loading

dsplaisted commented Jul 29, 2020

normj commented Jul 29, 2020

bonjo commented Jul 29, 2020

bjorg commented Jul 29, 2020

normj commented Jul 29, 2020

slavanap commented Aug 9, 2020

slavanap commented Aug 9, 2020

dsplaisted commented Aug 12, 2020

devssharma commented Aug 19, 2020

bjorg commented Aug 19, 2020

normj commented Oct 11, 2020

dsplaisted commented Oct 11, 2020

normj commented Oct 12, 2020

normj commented Mar 24, 2020 •

edited

Loading

rpopovych commented Apr 3, 2020 •

edited

Loading

davidruhmann commented Apr 10, 2020 •

edited

Loading

davidruhmann commented Apr 23, 2020 •

edited

Loading

bonjo commented Jul 29, 2020 •

edited

Loading