System.Runtime.Serialization.Formatters compatibility package question

[ericstj]

Opening this on behalf of @osexpert based on comment here dotnet/core#9726 (comment)

I have a problem:

Take the Linux Runtime Binaries x64
https://download.visualstudio.microsoft.com/download/pr/a15e92d8-e1db-402c-b06d-b6dcf7ad58eb/8c4233bceadcf8f57b40f64aceda69f7/dotnet-runtime-9.0.2-linux-x64.tar.gz

Inside there is System.Runtime.Serialization.Formatters.dll
In ILSpy:
[assembly: AssemblyVersion("8.1.0.0")]
[assembly: AssemblyFileVersion("9.0.225.6610")]
104KB

When publish non-self-contained, this file is being used instead if the one from here:
https://www.nuget.org/packages/System.Runtime.Serialization.Formatters/9.0.2
In ILSpy:
[assembly: AssemblyVersion("9.0.0.0")]
[assembly: AssemblyFileVersion("9.0.225.6610")]
133KB

This is all very confusing. Why do the release 9.0.2 include a file with assembly version 8.1.0.0?
And it would be fine if it would work, but it fails:
Could not load file or assembly 'System.Runtime.Serialization.Formatters, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

If manually copying the file from the nuget, it works.

Not sure if this is the right place to ask, it may not be related to this at all. Thanks.
Edit: maybe the runtime is ment to include an old versioned file, for some reason. Maybe the real problem is something with nuget, not overwriting/using the newer file from the included package, maybe because the package seems unused because of the BinaryFormatter magical mapping happening under the hood?
Edit2: I wish the System.Runtime.Serialization.Formatters.dll would contain something that the runtime did not, so it could be referenced by code to make it "stay" and not look completely unused...

[ericstj]

@osexpert I hope you don't mind, I copied your question here.

You can learn more about the NuGet package here https://learn.microsoft.com/en-us/dotnet/standard/serialization/binaryformatter-migration-guide/#use-the-compatibility-package

When using this package it will be copied to your output - it has a higher version than the framework package so that it's preferred over the framework version.

If you're not seeing this happen, you'll need to share a repro project so that we might have a look.

[osexpert]

I tried to make a minimal repro. At least I seem to have managed to here: https://github.com/osexpert/FormatterWrongVersionRepro

c:\test\FormatterWrongVersionRepro\src>dotnet publish -r linux-x64 -c Release --self-contained

c:\test\FormatterWrongVersionRepro\bin\Release\linux-x64\publish\System.Runtime.Serialization.Formatters.dll
114KB // wrong, it took the one from the runtime instead of the nuget
c:\test\FormatterWrongVersionRepro\bin\Release\System.Runtime.Serialization.Formatters.dll
133KB

As long as I only have App and Ext in the solution, it works. Adding an extra project and it seems to collapses.

So if you remove/delete ClassLibrary1 project, it suddenly start working and both places get the correct dll of 133KB.

Possibly I am doing something wrong, but I have never seen anything behave like this before.

A workaround may be to include a package reference to the System.Runtime.Serialization.Formatters nuget in every project, but this does not seems right...and hacks like this has never ever been needed before.

[chrarnoldus]

I think the problem in the repro is that all projects share the same output folder, even though they have conflicting dependencies. So when you run dotnet publish everything gets copied to the same folder and it is somewhat random which version of S.R.S.F.dll "wins".

[osexpert]

I think the problem in the repro is that all projects share the same output folder, even though they have conflicting dependencies. So when you run dotnet publish everything gets copied to the same folder and it is somewhat random which version of S.R.S.F.dll "wins".

Yes. But why does it work for everything else, and has done since beginning of time?

[ericstj]

You nailed it @chrarnoldus. You can specify /bl on the repro command to get a binlog, then open that with https://msbuildlog.com to see this problem:

This is caused because you're classlibrary is copying its output to the same place as the app and it has different dependencies, which is caused by hardcoding the output path to the same location for all projects:
https://github.com/osexpert/FormatterWrongVersionRepro/blob/b7c2fc927a8c9995a02be432f1e91d413325ce0f/src/Directory.Build.props#L8

why does it work for everything else, and has done since beginning of time?

It worked because you got lucky before - you never had projects which used different versions of the same library, so everything had the same dependencies and any case of clash was a noop. Now you've started using a package and only in some of your projects - so some projects get that new version others don't. You'd hit the same issue if you tried to use a newer System.Text.Json in some of your projects, or if you had projects referencing two different versions of any nuget package (regardless of framework overlap).

Dumping every project to the same folder is not a supported thing - as @chrarnoldus mentioned, nothing can resolve the clashing files. There are numerous issues in the SDK about this, dotnet/sdk#3886 for example. CC @dsplaisted

Your use of BaseOutputPath is broken for the same reason the SDK blocked output path with SLN. https://learn.microsoft.com/en-us/dotnet/core/compatibility/sdk/7.0/solution-level-output-no-longer-valid

PS - thank you for the excellent repro project @osexpert - this made it clear what was going on.

[osexpert]

I would understand it if I in fact had referenced the same nuget from different projects, but different versions.
But note: I have only references System.Runtime.Serialization.Formatters from one project/one version. And still a conflict/overwrite happens. And also see here:
https://learn.microsoft.com/en-us/dotnet/standard/serialization/binaryformatter-migration-guide/compatibility-package
"The only place that needs to depend on this package is the application project."
This is not really true then...because for this to work correctly now, is if all projects reference the nuget (this is the workaround I am currently using)

[osexpert]

"Dumping every project to the same folder is not a supported thing"
Yes...I guess I can only blame myself:-) So it sounds like this is by design then.

[ericstj]

I wouldn't say blame - the properties you're using to customize the build require some care to ensure those settings are safe to make. MSBuild allows you to change nearly anything, including things that can break your build in very unusual ways.

In addition to the error we mentioned here, a shared output can lead to flakiness during your build around files-in-use as well as build determinism issues. If you check out the link shared about the reason why the SDK removed the output path parameter for publish of SLN you can understand some of the implications of the shared output path.

If you'd like we could try to transfer this to MSBuild to see if it's possible to introduce a warning when multiple projects use the same output path, otherwise we can close this one.