Lock files prevent publishing when <RuntimeIdentifiers> is set

[bergbria]

Details about Problem

dotnet.exe --version (if appropriate): 2.2.300 and 3.0.100-preview5-011568

Automated repro

Download/unzip restore_multi_RID_repro.zip and run repro.bat

Detailed repro steps so we can see the same problem

1. Create a csproj with multiple RIDs specified in <RuntimeIdentifiers> (e.g. <RuntimeIdentifiers>osx-x64;win-x64</RuntimeIdentifiers>)

2. dotnet restore --use-lock-file

At this point, simply running dotnet publish won't actually create self-contained applications (e.g. a .exe file for windows). This is annoying, but not really Nuget's fault.

Normally, this would be resolved by running dotnet publish -r win-x64. However, this will actually mutate the lock file that was just generated.

If I enable locked mode in the publish command (this really should be a top-level flag, btw) via dotnet publish -p:RestoreWithLockFile=true -p:RestoreLockedMode=true -r win-x64, this will fail with error NU1004: The packages lock file is inconsistent with the project dependencies so restore can't be run in locked mode

Effectively, publishing doesn't play nicely with lock files when multiple RuntimeIdentifiers are in the csproj.

Presumably there's something in the restore target that looks at the <RuntimeIdentifier> property and discards <RuntimeIdentifiers> if it's set.

[nkolev92]

Since packages can contain architecture and platform specific assemblies, rids affect restore and the packages downloaded.
https://docs.microsoft.com/en-us/nuget/create-packages/supporting-multiple-target-frameworks#architecture-specific-folders
Each rid/tfm combination needs to go through the asset selection/dependency resolution process.

The self contained publish is effectively changing the restore inputs, and since the intermediates are shared, restore fails.

Not sure what our long term strategy is, as that will involve SDK/CLI people (@livarcocc), but for now, as a workaround you can do the following;

1. Include all rids in the project file.
2. Publish with no restore. dotnet publish -r win-x64 --no-restore in this case would just use the generated target graph for that rid in the full multi rid restores. Since the results for the specific target are exactly the same, this does not cause any issues.

[bergbria]

@nkolev92 - this fails with the same error whether or not I pass --no-restore

1. dotnet restore --use-lock-file
2. dotnet publish -p:RestoreWithLockFile=true -p:RestoreLockedMode=true -r win-x64 --no-restore

[nkolev92]

edit: Not sure what that first sentence was, lol :D

Does the 2nd step fail?

That should not trigger restore at all. So I'd imagine there's something else going wrong.

[bergbria]

@nkolev92 - yes, I'm sure. I've captured a diagnostic .binlog of the failed build and am attaching it here. (Note that I had to change the extension to .zip to make github happy, You'll want to rename before opening). publish_multiRID.zip

It's specifically failing in the ResolvePackageAssets target, which is run due to ResolveLockFileReferences, which gets hooked into the dependsontargets for ResolvePackageDependenciesForBuild, which is depended on by ResolveAssemblyReferences. None of those seem to be conditional.

[nkolev92]

Due to legacy reasons the ResolveLockFileReferences target actually refers to the assets file.

So as expected the restore operaton itself is not running, however the SDK error seems weird.
Not sure why that would be happening.

Does dotnet build --no-restore work?

[bergbria]

It works if I run restore first; else not.

[nkolev92]

oh, my bad, that was unclear.

So you have to run 1 full restore with all the rids and then just publish for each rid individually.

[bergbria]

Summary of behavior:

1. publish -r --no-restore
   a. "NETSDK1061: The project was restored using Microsoft.NETCore.App version 2.1.0, but with current settings, version 2.1.10 would be used instead."
   b. manifests with or without nuget lock files
2. publish -r
   a. no error without nuget lock files
   b. with nuget lock files
   1. with -p:RestoreLockedMode=true
      a. error NU1004: The packages lock file is inconsistent with the project dependencies so restore can't be run in locked mode
   2. without that
      b. succeeds but overwrites packages.lock.json

#1 seems like a bug in the dotnet sdk. Logged dotnet/sdk#3383 for that.

[nkolev92]

We synced up offline and the list by Brian summarizes the challenges.

As far as #2 is, that'd be by design as far as NuGet is concerned, but it's not intuitive, so we need to increase visibility.

Related #8289

[dasMulli]

Btw some RuntimeIdentifiers guidance was removed from the docs due to issues with the different package references for FDD and SDD dotnet/docs#9239

[aortiz-msft]

@nkolev92 - Not sure what the scope of this work is.

[nkolev92]

There are certain packages that are RID specific. Especially ones related to self contained publish.

Publish usually tends to run a restore with a certain RuntimeIdentifier/s set.
That can often be different from the regular workflow and the packages used there, so given that the inputs are different, you can't really do it in lock mode.

The action item here is to either

• come up with a guidance on how to handle this scenarios (example, multiple lock files?) Do we even want that? No lock file? Is that bad?
• Tweak the lock file format to account for this scenario. May require the guidance to be changed as well

[epvanhouten]

@nkolev92 Is the current recommendation to restore with dotnet restore --force-evaluate --runtime <RUNTIME_IDENTIFIER>?

[nkolev92]

@epvanhouten

We don't really have a well fleshed out recommendation at this point.
Potentially you could generate 2 lock files, and when you run publish, you can set NuGetLockFilePath to the extra lock file.

I wouldn't recommend just running force-evaluate as then you're defeating the locking.

[epvanhouten]

@nkolev92

Thanks for the guidance. I've added the following to my Directory.Build.props and seem to be getting satisfactory results when using dotnet publish --runtime linux-x64 --no-selfcontained -p:PublishReadyToRun=true.

<PropertyGroup Condition="$(RuntimeIdentifier) != ''" >
  <NuGetLockFilePath>packages.$(RuntimeIdentifier).lock.json</NuGetLockFilePath>
</PropertyGroup>

Are there any risks with doing this that I am missing?

[nkolev92]

As long as you make RuntimeIdentifier(s) defaults are not set, then I don't see any concerns.

The biggest challenge is remember to regenerate all lock files before pushing.

[rubo]

It's a pity that this is still an issue after being reported several years ago. Suggested workarounds and MSBuild configuration voodoos don't outweigh the underlying problem. They instead bring another bunch of headaches. A robust and viable solution is needed here; otherwise, package locking doesn't have much value with NuGet.