Dotnet Restore foo.sln -- fails when configurations in SLN cause duplicate (but diff config) projects in restore graph #4316

Closed
TheRealPiotrP opened this Issue Jan 18, 2017 · 33 comments

Comments

@TheRealPiotrP

Moving from dotnet/cli#4857 on behalf of @twsouthwick

Steps to reproduce

This solution has an ASP.NET Core project (targeting .NET 4.6) with a new csproj file and a number of libraries all targeting .NET 4.6 that are still using the old csproj format.

Run dotnet restore

I can build and run in Visual Studio 2017 RC, but fails at command line.

Expected behavior

Restores correctly

Actual behavior

  Restoring packages for C:\Users\*USER*\AppData\Local\Temp\u5vck04v.wg2\u5vck04v.wg2.csproj...
  Writing lock file to disk. Path: C:\Users\*USER*\AppData\Local\Temp\u5vck04v.wg2\obj\project.assets.json
  Generating MSBuild file C:\Users\*USER*\AppData\Local\Temp\u5vck04v.wg2\obj\u5vck04v.wg2.csproj.nuget.g.targets.
  Generating MSBuild file C:\Users\*USER*\AppData\Local\Temp\u5vck04v.wg2\obj\u5vck04v.wg2.csproj.nuget.g.props.
  Restore completed in 2382.5973ms for C:\Users\*USER*\AppData\Local\Temp\u5vck04v.wg2\u5vck04v.wg2.csproj.

  NuGet Config files used:
      C:\Users\*USER*\AppData\Roaming\NuGet\NuGet.Config
      C:\Program Files (x86)\NuGet\Config\Microsoft.VisualStudio.Offline.config

  Feeds used:
      C:\Users\*USER*\AppData\Local\Temp\0amxhhhs.f0v
c:\users\*USER*\Downloads\dotnet-dev-win-x64.latest\sdk\1.0.0-preview4-004130\NuGet.targets(70,5): error MSB4018: The "RestoreTask" task failed unexpectedly.\r [C:\Users\*USER*\Source\Repos\Project\Solution.sln]
c:\users\*USER*\Downloads\dotnet-dev-win-x64.latest\sdk\1.0.0-preview4-004130\NuGet.targets(70,5): error MSB4018: System.InvalidOperationException: Sequence contains more than one matching element\r [C:\Users\*USER*\Source\Repos\Project\Solution.sln]
c:\users\*USER*\Downloads\dotnet-dev-win-x64.latest\sdk\1.0.0-preview4-004130\NuGet.targets(70,5): error MSB4018:    at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source, Func`2 predicate)\r [C:\Users\*USER*\Source\Repos\Project\Solution.sln]
c:\users\*USER*\Downloads\dotnet-dev-win-x64.latest\sdk\1.0.0-preview4-004130\NuGet.targets(70,5): error MSB4018:    at NuGet.Commands.MSBuildRestoreUtility.GetPackageSpec(IEnumerable`1 items)\r [C:\Users\*USER*\Source\Repos\Project\Solution.sln]
c:\users\*USER*\Downloads\dotnet-dev-win-x64.latest\sdk\1.0.0-preview4-004130\NuGet.targets(70,5): error MSB4018:    at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()\r [C:\Users\*USER*\Source\Repos\Project\Solution.sln]
c:\users\*USER*\Downloads\dotnet-dev-win-x64.latest\sdk\1.0.0-preview4-004130\NuGet.targets(70,5): error MSB4018:    at NuGet.Commands.MSBuildRestoreUtility.GetDependencySpec(IEnumerable`1 items)\r [C:\Users\*USER*\Source\Repos\Project\Solution.sln]
c:\users\*USER*\Downloads\dotnet-dev-win-x64.latest\sdk\1.0.0-preview4-004130\NuGet.targets(70,5): error MSB4018:    at NuGet.Build.Tasks.RestoreTask.Execute()\r [C:\Users\*USER*\Source\Repos\Project\Solution.sln]
c:\users\*USER*\Downloads\dotnet-dev-win-x64.latest\sdk\1.0.0-preview4-004130\NuGet.targets(70,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()\r [C:\Users\*USER*\Source\Repos\Project\Solution.sln]
c:\users\*USER*\Downloads\dotnet-dev-win-x64.latest\sdk\1.0.0-preview4-004130\NuGet.targets(70,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.<ExecuteInstantiatedTask>d__25.MoveNext() [C:\Users\*USER*\Source\Repos\Project\Solution.sln]

Environment data

dotnet --info output:

.NET Command Line Tools (1.0.0-preview4-004130)

Product Information:
 Version:            1.0.0-preview4-004130
 Commit SHA-1 hash:  a0060ac321

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.10240
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   c:\users\*USER*\Downloads\dotnet-dev-win-x64.latest\sdk\1.0.0-preview4-004130

@rrelyea rrelyea added this to the 4.0 RTM milestone Jan 18, 2017

@rrelyea rrelyea added the Type:Bug label Jan 18, 2017

@rrelyea

This comment has been minimized.

Show comment
Hide comment
@rrelyea

rrelyea Jan 18, 2017

Contributor

@twsouthwick - do you have a sample project that shows this problem?

Contributor

rrelyea commented Jan 18, 2017

@twsouthwick - do you have a sample project that shows this problem?

@twsouthwick

This comment has been minimized.

Show comment
Hide comment
@twsouthwick

twsouthwick Jan 18, 2017

I don't have one at the moment. If I remember, I ran dotnet restore against the .sln and got this error. When I ran it against the .csproj for the specific project, it restore correctly. If this description isn't enough to get a repro, I can try to put one together.

I don't have one at the moment. If I remember, I ran dotnet restore against the .sln and got this error. When I ran it against the .csproj for the specific project, it restore correctly. If this description isn't enough to get a repro, I can try to put one together.

@emgarten

This comment has been minimized.

Show comment
Hide comment
@emgarten

emgarten Jan 19, 2017

Contributor

It looks like something is auto generating a project and referencing it, and that project doesn't contain valid properties for nuget.

@twsouthwick a repro would be helpful

Contributor

emgarten commented Jan 19, 2017

It looks like something is auto generating a project and referencing it, and that project doesn't contain valid properties for nuget.

@twsouthwick a repro would be helpful

@rrelyea

This comment has been minimized.

Show comment
Hide comment
@rrelyea

rrelyea Jan 25, 2017

Contributor

didn't repro the invalid operation exception at all, either.

dotnet restore is not aware of Packages.config.
@twsouthwick - Is that the scenario you ran into. If so, please use Nuget.exe restore instead.

Contributor

rrelyea commented Jan 25, 2017

didn't repro the invalid operation exception at all, either.

dotnet restore is not aware of Packages.config.
@twsouthwick - Is that the scenario you ran into. If so, please use Nuget.exe restore instead.

@rrelyea

This comment has been minimized.

Show comment
Hide comment
@rrelyea

rrelyea Jan 26, 2017

Contributor

Closing as not repro. @twsouthwick - would love more data or validation when you can.

Contributor

rrelyea commented Jan 26, 2017

Closing as not repro. @twsouthwick - would love more data or validation when you can.

@rrelyea rrelyea closed this Jan 26, 2017

@swoog

This comment has been minimized.

Show comment
Hide comment

swoog commented Jan 31, 2017

I have the same issue on the project https://github.com/swoog/Pattern/tree/develop

@emgarten

This comment has been minimized.

Show comment
Hide comment
@emgarten

emgarten Jan 31, 2017

Contributor

@swoog what version of dotnet CLI are you using?

Contributor

emgarten commented Jan 31, 2017

@swoog what version of dotnet CLI are you using?

@swoog

This comment has been minimized.

Show comment
Hide comment
@swoog

swoog Feb 2, 2017

The same as 1.0.0-preview4-004130

swoog commented Feb 2, 2017

The same as 1.0.0-preview4-004130

@natemcmaster

This comment has been minimized.

Show comment
Hide comment
@natemcmaster

natemcmaster Feb 6, 2017

@emgarten I ran into this as well using today's nightly build of CLI, 1.0.0-rc4-004769 on aspnet/Configuration.

Repro: unzip and run "dotnet restore" in the solution directory.

https://github.com/aspnet/Configuration/tree/2519ffc7fc1071befeae021551c6203126e117d3

Source: Configuration-2519ffc7fc1071befeae021551c6203126e117d3.zip

@emgarten I ran into this as well using today's nightly build of CLI, 1.0.0-rc4-004769 on aspnet/Configuration.

Repro: unzip and run "dotnet restore" in the solution directory.

https://github.com/aspnet/Configuration/tree/2519ffc7fc1071befeae021551c6203126e117d3

Source: Configuration-2519ffc7fc1071befeae021551c6203126e117d3.zip

@natemcmaster

This comment has been minimized.

Show comment
Hide comment
@natemcmaster

natemcmaster Feb 6, 2017

Also, WriteRestoreGraphTask fails with the same error:

  C:\Program Files (x86)\Microsoft Visual Studio\2017d15rel\Enterprise\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuG
et.targets(127,5): error MSB4018: The "WriteRestoreGraphTask" task failed unexpectedly.\r [C:\dev\Universe\Configuratio
n\Configuration.sln]
C:\Program Files (x86)\Microsoft Visual Studio\2017d15rel\Enterprise\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet
.targets(127,5): error MSB4018: System.InvalidOperationException: Sequence contains more than one matching element\r [C
:\dev\Universe\Configuration\Configuration.sln]
C:\Program Files (x86)\Microsoft Visual Studio\2017d15rel\Enterprise\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet
.targets(127,5): error MSB4018:    at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source, Func`2 pred
icate)\r [C:\dev\Universe\Configuration\Configuration.sln]
C:\Program Files (x86)\Microsoft Visual Studio\2017d15rel\Enterprise\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet
.targets(127,5): error MSB4018:    at NuGet.Commands.MSBuildRestoreUtility.GetPackageSpec(IEnumerable`1 items)\r [C:\de
v\Universe\Configuration\Configuration.sln]

Also, WriteRestoreGraphTask fails with the same error:

  C:\Program Files (x86)\Microsoft Visual Studio\2017d15rel\Enterprise\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuG
et.targets(127,5): error MSB4018: The "WriteRestoreGraphTask" task failed unexpectedly.\r [C:\dev\Universe\Configuratio
n\Configuration.sln]
C:\Program Files (x86)\Microsoft Visual Studio\2017d15rel\Enterprise\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet
.targets(127,5): error MSB4018: System.InvalidOperationException: Sequence contains more than one matching element\r [C
:\dev\Universe\Configuration\Configuration.sln]
C:\Program Files (x86)\Microsoft Visual Studio\2017d15rel\Enterprise\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet
.targets(127,5): error MSB4018:    at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source, Func`2 pred
icate)\r [C:\dev\Universe\Configuration\Configuration.sln]
C:\Program Files (x86)\Microsoft Visual Studio\2017d15rel\Enterprise\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet
.targets(127,5): error MSB4018:    at NuGet.Commands.MSBuildRestoreUtility.GetPackageSpec(IEnumerable`1 items)\r [C:\de
v\Universe\Configuration\Configuration.sln]
@natemcmaster

This comment has been minimized.

Show comment
Hide comment
@natemcmaster

natemcmaster Feb 6, 2017

I ran a structured logger and found what causes this line of code to throw.
Microsoft.Extensions.Configuration.Json.csproj ends up in RestoreGraphItems multiple times as 'Type=ProjectSpec', once with TargetFrameworks = net451;netstandard1.3 and again with TargetFrameworks = netstandard1.1.

You can view this with https://github.com/KirillOsenkov/MSBuildStructuredLog.
Log: restore.buildlog.txt

I ran a structured logger and found what causes this line of code to throw.
Microsoft.Extensions.Configuration.Json.csproj ends up in RestoreGraphItems multiple times as 'Type=ProjectSpec', once with TargetFrameworks = net451;netstandard1.3 and again with TargetFrameworks = netstandard1.1.

You can view this with https://github.com/KirillOsenkov/MSBuildStructuredLog.
Log: restore.buildlog.txt

@rrelyea

This comment has been minimized.

Show comment
Hide comment
@rrelyea

rrelyea Feb 6, 2017

Contributor

@emgarten - if we have a repro now, would love to know what the problem is...what the impact is...and what the fix would be.

Contributor

rrelyea commented Feb 6, 2017

@emgarten - if we have a repro now, would love to know what the problem is...what the impact is...and what the fix would be.

@rrelyea rrelyea reopened this Feb 6, 2017

@rrelyea

This comment has been minimized.

Show comment
Hide comment
@rrelyea

rrelyea Feb 6, 2017

Contributor

Marking Shiproom Required, as Nate/Eilon think this is blocking -- seeing errors migrating aspnet/configuration.

Contributor

rrelyea commented Feb 6, 2017

Marking Shiproom Required, as Nate/Eilon think this is blocking -- seeing errors migrating aspnet/configuration.

@emgarten

This comment has been minimized.

Show comment
Hide comment
@emgarten

emgarten Feb 6, 2017

Contributor

@rrelyea taking a look now.

Contributor

emgarten commented Feb 6, 2017

@rrelyea taking a look now.

@emgarten

This comment has been minimized.

Show comment
Hide comment
@emgarten

emgarten Feb 7, 2017

Contributor

I am able to repro this with @natemcmaster's https://github.com/aspnet/Configuration/tree/2519ffc7fc1071befeae021551c6203126e117d3

The repro works both on both

  • desktop MSBuild - VS 2017 2/05 d15rel
  • dotnet CLI MSBuild

The project Microsoft.Extensions.Configuration.csproj generates a ProjectSpec twice, both are the same from what I see, but they have a different randomly generated guid for the Include property. I verified all input properties to the <MSBuild> call are the same. I don't think there is an issue in NuGet.Targets.

It appears that MSBuild may not be caching the output of this target, and the result is that the duplicate project information is being emitted under different msbuild item include values.

I've narrowed when this broke down to:

  • Restore Success dotnet CLI 1.0.0-rc4-004652 - msbuild 15.1.523.56541
  • Restore Fails dotnet CLI 1.0.0-rc4-004655 - msbuild 15.1.539.38876

I don't see any .targets or .props changes that could have caused this when diffing the CLI folders between these two builds.

//cc @rainersigwald @jeffkl any idea on what might have caused this?

Contributor

emgarten commented Feb 7, 2017

I am able to repro this with @natemcmaster's https://github.com/aspnet/Configuration/tree/2519ffc7fc1071befeae021551c6203126e117d3

The repro works both on both

  • desktop MSBuild - VS 2017 2/05 d15rel
  • dotnet CLI MSBuild

The project Microsoft.Extensions.Configuration.csproj generates a ProjectSpec twice, both are the same from what I see, but they have a different randomly generated guid for the Include property. I verified all input properties to the <MSBuild> call are the same. I don't think there is an issue in NuGet.Targets.

It appears that MSBuild may not be caching the output of this target, and the result is that the duplicate project information is being emitted under different msbuild item include values.

I've narrowed when this broke down to:

  • Restore Success dotnet CLI 1.0.0-rc4-004652 - msbuild 15.1.523.56541
  • Restore Fails dotnet CLI 1.0.0-rc4-004655 - msbuild 15.1.539.38876

I don't see any .targets or .props changes that could have caused this when diffing the CLI folders between these two builds.

//cc @rainersigwald @jeffkl any idea on what might have caused this?

emgarten added a commit to NuGet/NuGet.Client that referenced this issue Feb 7, 2017

Handle duplicate MSBuild items during restore
This fix improves the robustness of Restore, allowing it to gracefully handle when MSBuild generates duplicate items for the same project.

Fixes NuGet/Home#4316

@emgarten emgarten referenced this issue in NuGet/NuGet.Client Feb 7, 2017

Closed

Handle duplicate MSBuild items during restore #1176

@rainersigwald

This comment has been minimized.

Show comment
Hide comment
@rainersigwald

rainersigwald Feb 8, 2017

I applied https://github.com/rainersigwald/ParallelBuildDebuggingLogger to debug this, scoping the query to the _GenerateRestoreProjectSpec target.

That showed that it was run twice for the same project:

Building target '_GenerateRestoreProjectSpec' in {6: "s:\work\restore-multi-instance\src\Microsoft.Extensions.Configuration\Microsoft.Extensions.Configuration.csproj" + <>}
Building target '_GenerateRestoreProjectSpec' in {71: "s:\work\restore-multi-instance\src\Microsoft.Extensions.Configuration\Microsoft.Extensions.Configuration.csproj" + <>}

The empty <> indicates that the traversal mechanism was doing the right thing in both cases and not setting any new global properties--so the problem (the distinct global properties) is inherited. Let's look at what instantiated projects 6 and 71, back to the root:

Project {2: "s:\work\restore-multi-instance\Configuration.sln" + <>} built by project -1 -- targets 'Restore'

Project {3: "s:\work\restore-multi-instance\src\Microsoft.Extensions.Configuration.Json\Microsoft.Extensions.Configuration.Json.csproj" + <NuGetRestoreTargets = C:\Program Files\dotnet\sdk\1.0.0-rc4-004771\NuGet.targets; Platform = AnyCPU; BuildProjectReferences = false; RestoreUseCustomAfterTargets = ; ExcludeRestorePackageImports = true; Configuration = Debug>} built by project 2 -- targets '_GenerateRestoreGraphProjectEntry'

Project {6: "s:\work\restore-multi-instance\src\Microsoft.Extensions.Configuration\Microsoft.Extensions.Configuration.csproj" + <>} built by project 3 -- targets '_GenerateRestoreGraphWalk'

Project {68: "s:\work\restore-multi-instance\src\Microsoft.Extensions.Configuration.DockerSecrets\Microsoft.Extensions.Configuration.DockerSecrets.csproj" + <NuGetRestoreTargets = C:\Program Files\dotnet\sdk\1.0.0-rc4-004771\NuGet.targets; Platform = x86; BuildProjectReferences = false; RestoreUseCustomAfterTargets = ; ExcludeRestorePackageImports = true; Configuration = Debug>} built by project 2 -- targets '_GenerateRestoreGraphProjectEntry'

Project {71: "s:\work\restore-multi-instance\src\Microsoft.Extensions.Configuration\Microsoft.Extensions.Configuration.csproj" + <>} built by project 68 -- targets '_GenerateRestoreGraphWalk'

So the solution (2) builds Microsoft.Extensions.Configuration.Json (3) as AnyCPU and Microsoft.Extensions.Configuration.DockerSecrets (68) as x86. Those each have a reference to Microsoft.Extensions.Configuration, so it gets built twice (6 as AnyCPU and 71 as x86).

I applied https://github.com/rainersigwald/ParallelBuildDebuggingLogger to debug this, scoping the query to the _GenerateRestoreProjectSpec target.

That showed that it was run twice for the same project:

Building target '_GenerateRestoreProjectSpec' in {6: "s:\work\restore-multi-instance\src\Microsoft.Extensions.Configuration\Microsoft.Extensions.Configuration.csproj" + <>}
Building target '_GenerateRestoreProjectSpec' in {71: "s:\work\restore-multi-instance\src\Microsoft.Extensions.Configuration\Microsoft.Extensions.Configuration.csproj" + <>}

The empty <> indicates that the traversal mechanism was doing the right thing in both cases and not setting any new global properties--so the problem (the distinct global properties) is inherited. Let's look at what instantiated projects 6 and 71, back to the root:

Project {2: "s:\work\restore-multi-instance\Configuration.sln" + <>} built by project -1 -- targets 'Restore'

Project {3: "s:\work\restore-multi-instance\src\Microsoft.Extensions.Configuration.Json\Microsoft.Extensions.Configuration.Json.csproj" + <NuGetRestoreTargets = C:\Program Files\dotnet\sdk\1.0.0-rc4-004771\NuGet.targets; Platform = AnyCPU; BuildProjectReferences = false; RestoreUseCustomAfterTargets = ; ExcludeRestorePackageImports = true; Configuration = Debug>} built by project 2 -- targets '_GenerateRestoreGraphProjectEntry'

Project {6: "s:\work\restore-multi-instance\src\Microsoft.Extensions.Configuration\Microsoft.Extensions.Configuration.csproj" + <>} built by project 3 -- targets '_GenerateRestoreGraphWalk'

Project {68: "s:\work\restore-multi-instance\src\Microsoft.Extensions.Configuration.DockerSecrets\Microsoft.Extensions.Configuration.DockerSecrets.csproj" + <NuGetRestoreTargets = C:\Program Files\dotnet\sdk\1.0.0-rc4-004771\NuGet.targets; Platform = x86; BuildProjectReferences = false; RestoreUseCustomAfterTargets = ; ExcludeRestorePackageImports = true; Configuration = Debug>} built by project 2 -- targets '_GenerateRestoreGraphProjectEntry'

Project {71: "s:\work\restore-multi-instance\src\Microsoft.Extensions.Configuration\Microsoft.Extensions.Configuration.csproj" + <>} built by project 68 -- targets '_GenerateRestoreGraphWalk'

So the solution (2) builds Microsoft.Extensions.Configuration.Json (3) as AnyCPU and Microsoft.Extensions.Configuration.DockerSecrets (68) as x86. Those each have a reference to Microsoft.Extensions.Configuration, so it gets built twice (6 as AnyCPU and 71 as x86).

@rrelyea

This comment has been minimized.

Show comment
Hide comment
@rrelyea

rrelyea Feb 8, 2017

Contributor

@rainersigwald - so what does that mean?

Contributor

rrelyea commented Feb 8, 2017

@rainersigwald - so what does that mean?

@natemcmaster

This comment has been minimized.

Show comment
Hide comment
@natemcmaster

natemcmaster Feb 8, 2017

NuGet fails because the solution has "x86", "x64", and "Any CPU" solution configurations.

I can confirm that removing the x86 and x64 from Configuration.sln solves the issue. It would still be good to harden NuGet to handle this better.

NuGet fails because the solution has "x86", "x64", and "Any CPU" solution configurations.

I can confirm that removing the x86 and x64 from Configuration.sln solves the issue. It would still be good to harden NuGet to handle this better.

@natemcmaster natemcmaster referenced this issue in aspnet/Configuration Feb 8, 2017

Merged

Remove x86 and x64 solution configurations #593

@rrelyea

This comment has been minimized.

Show comment
Hide comment
@rrelyea

rrelyea Feb 8, 2017

Contributor

so are we saying without this fix, anytime we have anycpu plus any other config, we will fail in restore?

Contributor

rrelyea commented Feb 8, 2017

so are we saying without this fix, anytime we have anycpu plus any other config, we will fail in restore?

@rrelyea

This comment has been minimized.

Show comment
Hide comment
@rrelyea

rrelyea Feb 8, 2017

Contributor

Also...is this caused to be more prevalent by: dotnet/project-system#1458
When they fix that, will this be less important?

Contributor

rrelyea commented Feb 8, 2017

Also...is this caused to be more prevalent by: dotnet/project-system#1458
When they fix that, will this be less important?

@rainersigwald

This comment has been minimized.

Show comment
Hide comment
@rainersigwald

rainersigwald Feb 8, 2017

From talking to @emgarten there are some conflicting requirements here:

  • A project can build for different platforms. These produce distinct outputs and thus must be built (and evaluated) separately.
  • A project can only have one assets file.

My understanding is that the restore traversal already considers TFM as a dimension (and unifies references that are conditional on TFM), but doesn't consider configuration and platform. That makes sense in the new world but in the old world, sln builds care deeply about config/platform.

I can see a couple of options here:

  • Remove platform from the p2p ref resolved during Restore. This would get the default platform for the projects (at least for new projects). If the user was doing something tricky with platforms, this might produce incorrect results.
  • Deduplicate/unify the references. Ideally have an additional dimension in the assets file for configuration and one for platform, but that may not be fully necessary.

The latter seems generally simpler, but I don't understand the NuGet implications deeply.

is this caused to be more prevalent by: dotnet/project-system#1458

That's a good question and the answer isn't clear to me at the moment. I'll try to drill in a bit more in the morning.

From talking to @emgarten there are some conflicting requirements here:

  • A project can build for different platforms. These produce distinct outputs and thus must be built (and evaluated) separately.
  • A project can only have one assets file.

My understanding is that the restore traversal already considers TFM as a dimension (and unifies references that are conditional on TFM), but doesn't consider configuration and platform. That makes sense in the new world but in the old world, sln builds care deeply about config/platform.

I can see a couple of options here:

  • Remove platform from the p2p ref resolved during Restore. This would get the default platform for the projects (at least for new projects). If the user was doing something tricky with platforms, this might produce incorrect results.
  • Deduplicate/unify the references. Ideally have an additional dimension in the assets file for configuration and one for platform, but that may not be fully necessary.

The latter seems generally simpler, but I don't understand the NuGet implications deeply.

is this caused to be more prevalent by: dotnet/project-system#1458

That's a good question and the answer isn't clear to me at the moment. I'll try to drill in a bit more in the morning.

@rainersigwald

This comment has been minimized.

Show comment
Hide comment
@rainersigwald

rainersigwald Feb 8, 2017

is this caused to be more prevalent by: dotnet/project-system#1458

I believe the answer is "no", but it might be indirectly related. Looking at the Configuration repro, the issue is caused because the default platform for the DockerSecrets projects is set to x86:
image

I changed that to Any CPU, matching the other projects, and saw a successful dotnet restore (without going so far as removing the solution configurations as in aspnet/Configuration#593).

Diff of .sln after changing "Mixed Platforms" to point to `Any CPU` for DockerSecrets
diff --git a/Configuration.sln b/Configuration.sln
index bc913b0..af124b2 100644
--- a/Configuration.sln
+++ b/Configuration.sln
@@ -1,6 +1,6 @@
 Microsoft Visual Studio Solution File, Format Version 12.00
 # Visual Studio 15
-VisualStudioVersion = 15.0.26118.1
+VisualStudioVersion = 15.0.26206.0
 MinimumVisualStudioVersion = 10.0.40219.1
 Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{F141E2D0-F9B8-4ADB-A19A-7B6FF4CA19A1}"
 EndProject
@@ -444,8 +444,8 @@ Global
                {AC7FAD2A-5763-404D-B0FC-3CCA81A16B0A}.Release|x86.Build.0 = Release|Any CPU
                {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
                {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|Any CPU.Build.0 = Debug|Any CPU
-               {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|Mixed Platforms.ActiveCfg = Debug|x86
-               {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|Mixed Platforms.Build.0 = Debug|x86
+               {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU
+               {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU
                {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|x64.ActiveCfg = Debug|x64
                {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|x64.Build.0 = Debug|x64
                {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|x86.ActiveCfg = Debug|x86
@@ -460,8 +460,8 @@ Global
                {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Release|x86.Build.0 = Release|x86
                {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
                {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|Any CPU.Build.0 = Debug|Any CPU
-               {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|Mixed Platforms.ActiveCfg = Debug|x86
-               {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|Mixed Platforms.Build.0 = Debug|x86
+               {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU
+               {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU
                {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|x64.ActiveCfg = Debug|x64
                {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|x64.Build.0 = Debug|x64
                {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|x86.ActiveCfg = Debug|x86

@HaoK do you recall if you did anything special or interesting regarding configurations when you added the DockerSecrets stuff in aspnet/Configuration@2519ffc? If you didn't, whatever caused the default solution configuration to point to the x86 platform builds of those projects is both wrong on principle and exacerbating this problem.

rainersigwald commented Feb 8, 2017

is this caused to be more prevalent by: dotnet/project-system#1458

I believe the answer is "no", but it might be indirectly related. Looking at the Configuration repro, the issue is caused because the default platform for the DockerSecrets projects is set to x86:
image

I changed that to Any CPU, matching the other projects, and saw a successful dotnet restore (without going so far as removing the solution configurations as in aspnet/Configuration#593).

Diff of .sln after changing "Mixed Platforms" to point to `Any CPU` for DockerSecrets
diff --git a/Configuration.sln b/Configuration.sln
index bc913b0..af124b2 100644
--- a/Configuration.sln
+++ b/Configuration.sln
@@ -1,6 +1,6 @@
 Microsoft Visual Studio Solution File, Format Version 12.00
 # Visual Studio 15
-VisualStudioVersion = 15.0.26118.1
+VisualStudioVersion = 15.0.26206.0
 MinimumVisualStudioVersion = 10.0.40219.1
 Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{F141E2D0-F9B8-4ADB-A19A-7B6FF4CA19A1}"
 EndProject
@@ -444,8 +444,8 @@ Global
                {AC7FAD2A-5763-404D-B0FC-3CCA81A16B0A}.Release|x86.Build.0 = Release|Any CPU
                {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
                {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|Any CPU.Build.0 = Debug|Any CPU
-               {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|Mixed Platforms.ActiveCfg = Debug|x86
-               {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|Mixed Platforms.Build.0 = Debug|x86
+               {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU
+               {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU
                {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|x64.ActiveCfg = Debug|x64
                {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|x64.Build.0 = Debug|x64
                {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Debug|x86.ActiveCfg = Debug|x86
@@ -460,8 +460,8 @@ Global
                {69AB0230-D82E-438B-AFE5-85BFF414F1B8}.Release|x86.Build.0 = Release|x86
                {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
                {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|Any CPU.Build.0 = Debug|Any CPU
-               {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|Mixed Platforms.ActiveCfg = Debug|x86
-               {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|Mixed Platforms.Build.0 = Debug|x86
+               {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU
+               {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU
                {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|x64.ActiveCfg = Debug|x64
                {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|x64.Build.0 = Debug|x64
                {82A403ED-F827-4FED-BE38-7F87925A07E1}.Debug|x86.ActiveCfg = Debug|x86

@HaoK do you recall if you did anything special or interesting regarding configurations when you added the DockerSecrets stuff in aspnet/Configuration@2519ffc? If you didn't, whatever caused the default solution configuration to point to the x86 platform builds of those projects is both wrong on principle and exacerbating this problem.

@rainersigwald

This comment has been minimized.

Show comment
Hide comment
@rainersigwald

rainersigwald Feb 8, 2017

Oh, and the answer to "what changed between MSBuild 15.1.523.56541 and 15.1.539.38876 to expose this?" is Microsoft/msbuild#1590, which fixed a bug that was causing solution-level restore to not run, in favor of an inefficient jumble of project-level restore invocations.

Oh, and the answer to "what changed between MSBuild 15.1.523.56541 and 15.1.539.38876 to expose this?" is Microsoft/msbuild#1590, which fixed a bug that was causing solution-level restore to not run, in favor of an inefficient jumble of project-level restore invocations.

@rainersigwald

This comment has been minimized.

Show comment
Hide comment
@rainersigwald

rainersigwald Feb 8, 2017

I just tried adding a new library to the Configuration sln and it was added as x86, presumably because of dotnet/project-system#1458. So that would indeed exacerbate the problem here--without that, prevalence of solution configurations specifying platform in new projects should be very low.

I just tried adding a new library to the Configuration sln and it was added as x86, presumably because of dotnet/project-system#1458. So that would indeed exacerbate the problem here--without that, prevalence of solution configurations specifying platform in new projects should be very low.

@attilah

This comment has been minimized.

Show comment
Hide comment
@attilah

attilah Feb 8, 2017

I was just hit by this issue and having Any CPU and x64 for the same build configuration caused it, however it is a fully legal configuration, specially if one of the projects depending on some nugets which has platform specific binaries.

The funny thing that it was only failing from dotnet.exe command line, VS 2017 RC4 did something differently and handled it somehow.

attilah commented Feb 8, 2017

I was just hit by this issue and having Any CPU and x64 for the same build configuration caused it, however it is a fully legal configuration, specially if one of the projects depending on some nugets which has platform specific binaries.

The funny thing that it was only failing from dotnet.exe command line, VS 2017 RC4 did something differently and handled it somehow.

@rrelyea

This comment has been minimized.

Show comment
Hide comment
@rrelyea

rrelyea Feb 8, 2017

Contributor

@attilah - thanks for the "+1" -- yes, today nuget restore has this problem, but restore in VS and NuGet.exe both avoid this problem.

Contributor

rrelyea commented Feb 8, 2017

@attilah - thanks for the "+1" -- yes, today nuget restore has this problem, but restore in VS and NuGet.exe both avoid this problem.

@rrelyea rrelyea changed the title from RestoreTask fails with InvalidOperationException to Dotnet Restore foo.sln -- fails when configurations in SLN cause duplicate (but diff config) projects in restore graph Feb 8, 2017

@iskiselev

This comment has been minimized.

Show comment
Hide comment
@iskiselev

iskiselev Feb 9, 2017

@rrelyea, is nuget.exe 4.0 publicly accessible somewhere to workaround this problem?

@rrelyea, is nuget.exe 4.0 publicly accessible somewhere to workaround this problem?

@onovotny

This comment has been minimized.

Show comment
Hide comment
@onovotny

onovotny Feb 9, 2017

I'm still hitting this in RC4...and it's blocking me with xUnit for Devices:
https://github.com/xunit/devices.xunit/blob/master/xUnit.Devices.sln

I'm trying to call msbuild /t:restore xunit.devices.sln and get:

C:\dev\git\devices.xunit [master ≡]> msbuild /t:restore .\xUnit.Devices.sln
Microsoft (R) Build Engine version 15.1.545.13942
Copyright (C) Microsoft Corporation. All rights reserved.

Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
Build started 2/8/2017 4:18:22 PM.
Project "C:\dev\git\devices.xunit\xUnit.Devices.sln" on node 1 (restore target(s)).
ValidateSolutionConfiguration:
  Building solution configuration "Debug|Mixed Platforms".
ValidateProjects:
  The project "integration.iOS" is not selected for building in solution configuration "Debug|Mixed Platforms".
  The project "integration.android" is not selected for building in solution configuration "Debug|Mixed Platforms".
  The project "integration.uwp" is not selected for building in solution configuration "Debug|Mixed Platforms".
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet.targets(97,5):
error : Sequence contains more than one matching element [C:\dev\git\devices.xunit\xUnit.Devices.sln]
Done Building Project "C:\dev\git\devices.xunit\xUnit.Devices.sln" (restore target(s)) -- FAILED.


Build FAILED.

"C:\dev\git\devices.xunit\xUnit.Devices.sln" (restore target) (1) ->
(Restore target) ->
  C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet.targets(97,5)
: error : Sequence contains more than one matching element [C:\dev\git\devices.xunit\xUnit.Devices.sln]

    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:01.76

onovotny commented Feb 9, 2017

I'm still hitting this in RC4...and it's blocking me with xUnit for Devices:
https://github.com/xunit/devices.xunit/blob/master/xUnit.Devices.sln

I'm trying to call msbuild /t:restore xunit.devices.sln and get:

C:\dev\git\devices.xunit [master ≡]> msbuild /t:restore .\xUnit.Devices.sln
Microsoft (R) Build Engine version 15.1.545.13942
Copyright (C) Microsoft Corporation. All rights reserved.

Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
Build started 2/8/2017 4:18:22 PM.
Project "C:\dev\git\devices.xunit\xUnit.Devices.sln" on node 1 (restore target(s)).
ValidateSolutionConfiguration:
  Building solution configuration "Debug|Mixed Platforms".
ValidateProjects:
  The project "integration.iOS" is not selected for building in solution configuration "Debug|Mixed Platforms".
  The project "integration.android" is not selected for building in solution configuration "Debug|Mixed Platforms".
  The project "integration.uwp" is not selected for building in solution configuration "Debug|Mixed Platforms".
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet.targets(97,5):
error : Sequence contains more than one matching element [C:\dev\git\devices.xunit\xUnit.Devices.sln]
Done Building Project "C:\dev\git\devices.xunit\xUnit.Devices.sln" (restore target(s)) -- FAILED.


Build FAILED.

"C:\dev\git\devices.xunit\xUnit.Devices.sln" (restore target) (1) ->
(Restore target) ->
  C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet.targets(97,5)
: error : Sequence contains more than one matching element [C:\dev\git\devices.xunit\xUnit.Devices.sln]

    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:01.76
@iskiselev

This comment has been minimized.

Show comment
Hide comment
@iskiselev

iskiselev Feb 9, 2017

I have the same problem, if configuration of any project doesn't match Solution configuration. So, not only setting Platform=x64 to some project within AnyCpu solution platform, but also setting Release configuration to some projects in Debug solution configuration provokes it.

I have the same problem, if configuration of any project doesn't match Solution configuration. So, not only setting Platform=x64 to some project within AnyCpu solution platform, but also setting Release configuration to some projects in Debug solution configuration provokes it.

emgarten added a commit to NuGet/NuGet.Client that referenced this issue Feb 9, 2017

Handle duplicate MSBuild items during restore
This fix improves the robustness of Restore, allowing it to gracefully handle when MSBuild generates duplicate items for the same project.

Fixes NuGet/Home#4316
@emgarten

This comment has been minimized.

Show comment
Hide comment
@emgarten

emgarten Feb 9, 2017

Contributor

Use nuget.exe to workaround this issue: https://dist.nuget.org/win-x86-commandline/v4.0.0-rc4/nuget.exe

Or Visual Studio as @rrelyea mentioned

Contributor

emgarten commented Feb 9, 2017

Use nuget.exe to workaround this issue: https://dist.nuget.org/win-x86-commandline/v4.0.0-rc4/nuget.exe

Or Visual Studio as @rrelyea mentioned

@rrelyea

This comment has been minimized.

Show comment
Hide comment
@rrelyea

rrelyea Feb 9, 2017

Contributor

Please finish PR review/feedback (if any).
Then ensure you've tested it all it needs.
Then merge into dev.

Contributor

rrelyea commented Feb 9, 2017

Please finish PR review/feedback (if any).
Then ensure you've tested it all it needs.
Then merge into dev.

@natemcmaster natemcmaster referenced this issue in aspnet/Performance Feb 9, 2017

Closed

Migrate repo to .csproj #206

@twsouthwick

This comment has been minimized.

Show comment
Hide comment
@twsouthwick

twsouthwick Feb 10, 2017

Can you post when this should end up in an available build? I've got a few projects hitting this and would like to test as soon as it's available.

Can you post when this should end up in an available build? I've got a few projects hitting this and would like to test as soon as it's available.

@kzu

This comment has been minimized.

Show comment
Hide comment
@kzu

kzu Feb 20, 2017

I'm seeing this in the latest VS2017RC too. Any ideas when this will hit the stable channel?

Repro: clone & run msbuild /t:restore in folder https://github.com/NuGet/NuGet.Build.Packaging/tree/9de80814eadc29d15b27c4a674139a53b71a2558/src/Build

Version information:

Microsoft Visual Studio Enterprise 2017 RC
Version 15.0.26206.0 D15REL
Microsoft .NET Framework
Version 4.6.01586

Installed Version: Enterprise
...
NuGet Package Manager   4.0.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.

kzu commented Feb 20, 2017

I'm seeing this in the latest VS2017RC too. Any ideas when this will hit the stable channel?

Repro: clone & run msbuild /t:restore in folder https://github.com/NuGet/NuGet.Build.Packaging/tree/9de80814eadc29d15b27c4a674139a53b71a2558/src/Build

Version information:

Microsoft Visual Studio Enterprise 2017 RC
Version 15.0.26206.0 D15REL
Microsoft .NET Framework
Version 4.6.01586

Installed Version: Enterprise
...
NuGet Package Manager   4.0.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.

kzu added a commit to kzu/ApiInteresect that referenced this issue Feb 20, 2017

Use AnyCPU as the target platform
This avoids an annoying NuGet bug (NuGet/Home#4316)
that prevents restore from `msbuild /t:restore`

@kzu kzu referenced this issue in xamarin/ApiInteresect Feb 20, 2017

Merged

Use AnyCPU as the target platform #1

kzu added a commit to kzu/ApiInteresect that referenced this issue Feb 20, 2017

Use AnyCPU as the target platform
This avoids an annoying NuGet bug (NuGet/Home#4316)
that prevents restore from `msbuild /t:restore`

mrward added a commit to xamarin/ApiInteresect that referenced this issue Feb 20, 2017

Use AnyCPU as the target platform
This avoids an annoying NuGet bug (NuGet/Home#4316)
that prevents restore from `msbuild /t:restore`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment