CoreFx implementation assemblies contain duplicate definitions of public types

[tmat]

The public types listed below have multiple definitions in CoreFX assemblies. This causes issues when the assemblies are used in dynamic/runtime compilation scenarios such as C# scripting (see e.g. https://github.com/dotnet/cli/issues/320), dynamic languages, dynamic runtime binders, etc. and in VS Expression Evaluators. All these tools work with implementation assemblies since the reference assemblies are not available.

These duplicate definitions should be made internal.

Duplicate types in recent .NET Core build:

Microsoft.Win32.SafeHandles.SafeRegistryHandle: "Microsoft.Win32.Registry, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"; "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"
Microsoft.Win32.SafeHandles.SafeFileHandle: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.IO.FileSystem, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.BitConverter: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Runtime.Extensions, Version=4.0.11.0, PKT=b03f5f7f11d50a3a"
System.Console: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Console, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.IO.Directory: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.IO.FileSystem, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.IO.DirectoryInfo: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.IO.FileSystem, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.IO.SearchOption: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.IO.FileSystem, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.IO.File: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.IO.FileSystem, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.IO.FileAccess: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.IO.FileSystem.Primitives, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.IO.FileInfo: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.IO.FileSystem, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.IO.FileMode: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.IO.FileSystem.Primitives, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.IO.FileShare: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.IO.FileSystem.Primitives, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.IO.FileStream: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.IO.FileSystem, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.IO.FileSystemInfo: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.IO.FileSystem, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.IO.FileAttributes: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.IO.FileSystem.Primitives, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.IO.Path: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Runtime.Extensions, Version=4.0.11.0, PKT=b03f5f7f11d50a3a"
System.Collections.CollectionBase: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Collections.NonGeneric, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.Collections.StructuralComparisons: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Collections, Version=4.0.11.0, PKT=b03f5f7f11d50a3a"
System.Collections.ObjectModel.ReadOnlyDictionary`2: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.ObjectModel, Version=4.0.11.0, PKT=b03f5f7f11d50a3a"
System.Collections.ObjectModel.KeyedCollection`2: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.ObjectModel, Version=4.0.11.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.CipherMode: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Primitives, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.PaddingMode: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Primitives, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.KeySizes: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Primitives, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.CryptographicException: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Primitives, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.ICryptoTransform: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Primitives, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.RandomNumberGenerator: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Algorithms, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.Aes: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Algorithms, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.AsymmetricAlgorithm: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Primitives, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.CspProviderFlags: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Csp, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.CspParameters: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Csp, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.CryptoStreamMode: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Primitives, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.CryptoStream: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Primitives, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.DeriveBytes: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Algorithms, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.HMAC: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Primitives, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.HMACSHA1: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Algorithms, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.HMACSHA256: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Algorithms, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.HMACSHA384: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Algorithms, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.HMACSHA512: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Algorithms, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.HashAlgorithm: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Primitives, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.KeyNumber: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Csp, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.CspKeyContainerInfo: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Csp, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.ICspAsymmetricAlgorithm: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Csp, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.KeyedHashAlgorithm: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Primitives, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.MD5: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Algorithms, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.Rfc2898DeriveBytes: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Algorithms, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.RSAParameters: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Algorithms, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.RSA: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Algorithms, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.RSACryptoServiceProvider: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Csp, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.SHA1: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Algorithms, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.SHA256: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Algorithms, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.SHA384: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Algorithms, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.SHA512: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Algorithms, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.SymmetricAlgorithm: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.Primitives, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.X509Certificates.X509ContentType: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.X509Certificates, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.X509Certificates.X509KeyStorageFlags: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.X509Certificates, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Cryptography.X509Certificates.X509Certificate: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Cryptography.X509Certificates, Version=4.0.0.0, PKT=b03f5f7f11d50a3a"
System.Security.Principal.IIdentity: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Principal, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.Security.Principal.IPrincipal: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Principal, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.Security.Principal.TokenImpersonationLevel: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Security.Principal, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.Threading.WaitHandleExtensions: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Runtime.Handles, Version=4.0.1.0, PKT=b03f5f7f11d50a3a"
System.Threading.CountdownEvent: "mscorlib, Version=4.0.0.0, PKT=7cec85d7bea7798e"; "System.Threading, Version=4.0.11.0, PKT=b03f5f7f11d50a3a"

[tmat]

@weshaggard @terrajobst

[tmat]

@weshaggard If we agree on adding the attribute, why not do it for .NET Core 1.0? If we don't do it we'll end up with assemblies that don't have these types marked and we will need to handle such cases forever in the debugger (EEs).

[weshaggard]

I'm not sure I want to create a new attribute (even if it is an internally defined one) and apply this to various types in our new libraries. Your assembly identity makes the assumption that they are only duplicated in CoreCLR's mscorlib but what about full .NET Framework or other platforms where the types are in different assemblies and these new libraries are supported there?

How do these technologies deal with duplicated types in other scenarios? I realize these make the duplication more common but there are still plenty of cases where people will run into duplicated types in the wild and we have to handle them in some way.

If you are primarily concerned with CoreCLR here can we not special case and prefer types that are not in mscorlib? Ultimately I would like to try and remove these duplicates in mscorlib but that isn't going to happen in our .NET Core v1.0 timeframe.

[tmat]

I was just scanning CoreCLR bits -- so all the listed duplicates are from mscorlib and one other library.

In C# duplicate types can be dealt with via extern aliases. Although not currently supported by C# scripts we could support them. However, extern aliases have rarely been needed in simple projects because duplication occurred rarely. Having to alias basic types like Console, Path, File, etc. is really bad experience.

I assume that whenever we decide to add a type to a System library that already exists in another implementation assembly on any platform we know what implementation assemblies it exists in. So my proposal is to mark the new type with SupersedesTypeDefinedInAssembly attributes -- one for each implementation assembly that has shipped in which it exists.

My concern is for all system libraries on both Desktop FX and CoreCLR where we decide for whatever reason that we need to create a new implementation for a type that is already implemented in another assembly instead of updating the existing definition.

[guardrex]

@weshaggard I see in the list above a duplicate type problem that I'm facing myself. I'm using ICryptoTransform with Data Protection for some encryption/decryption. I brought in "dotnet-test-xunit": "1.0.0-dev-45337-57" to do some testing with new xUnit bits ... 💥.

The type 'ICryptoTransform' exists in both 'System.Security.Cryptography.Encryption, 
Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' and 
'System.Security.Cryptography.Primitives, Version=4.0.0.0, Culture=neutral, 
PublicKeyToken=b03f5f7f11d50a3a'

I can provide a repro if you like (or project.lock.json). If you have any kind of workaround, that would be great. This is not a blocking issue for me, as I'm just testing xUnit bits. Otherwise, I'm going to strip out Data Protection from this test app to avoid the problem for now.

[bartonjs]

@guardrex System.Security.Cryptography.Encryption was deleted in... August? Something in that package chain is referencing a really old version of the crypto libraries.

[guardrex]

@bartonjs Tracing back from System.Security.Cryptography.Encryption in the lock file ...

System.Security.Cryptography.Encryption 4.0.0-beta-23109 -->
System.Security.SecureString 4.0.0-beta-23109 -->
System.Diagnostics.Process 4.0.0-beta-23109 -->
NuGet.Common 3.4.0-beta-559

This isn't a blocking issue for me, so if its best to wait, I can certainly do so. If you want a repro, let me know, and I'll have one up by this evening (or I can post the lock file if that's all you need).

[bartonjs]

@guardrex Thanks for the info. I've opened NuGet/Home#2181 to hopefully get NuGet.Common using newer references so it doesn't resurrect the dead S.S.C.Encryption package.

[vors]

This issue affects PowerShell Add-Type PowerShell/PowerShell#1616

[weshaggard]

@danmosemsft you were asking if we had an issue tracking this duplication, turns out we do have this one. You can also see the duplicated types in the netstandard facade generation as seed preferences (https://github.com/dotnet/standard/blob/wesh/netcoreapp_compat/netstandard/src/netstandard.csproj#L55).

@vors while this is definitely not desired there is a good chance we will never fully eliminate the duplication. As long as people only use APIs from the reference assemblies they should be OK but while looking at all the types in the implementation there will be conflicts.

[tmat]

@weshaggard Is it only matter of resources or is it inherently not fixable?

[jkotas]

@rahku is going to look into fixing this as part of dotnet/corefx#12678.

there is a good chance we will never fully eliminate the duplication

We should be able to fully eliminate the duplication of public types. The types may be still duplicated in some cases, but only one of them will be marked public - the poor duplicate will be internal.

[vors]

PowerShell team will very much appreciate if it could be fixed.
Our customers hit problems using Add-Type on alpha build because of the type duplications.
See PowerShell/PowerShell#1616 (comment) for the discussion.

@tmat helped me to create a reflection-based workaround PowerShell/PowerShell#2704 but the team rejected it, because well, reflection.

[tmat]

@jkotas Sounds great. Internal dups are fine.

[ReubenBond]

We ran into this issue with our runtime code generation in dotnet/orleans while porting to .NET Core/Standard. Currently I'm trying the reflection-based hack implemented by @vors for PowerShell.

[jdom]

This workaround worked fine for us in Orleans, so we are using it until this gets exposed publicly. In case you are interested, the merged PR is here: dotnet/orleans#2466

[vors]

@jdom glad that you found the work-aroubd useful. To double-check that we are all on the same page: there is no commitment from @tmat (Roslyn) that this API will ever be public. Instead, the proposition is to remove public types in CoreFX (this issue).

[jdom]

I see, well, good enough too. Hopefully that reflection hack will still work, but anyway we can stop using the hack altogether and just reference a newer version of the standard once that happens. A ping in this thread with a heads up would be valuable if possible when something changes regarding this.

[danmoseley]

I'm going to close this in favor of dotnet/corefx#12678 and https://github.com/dotnet/coreclr/issues/4651 to bring a little clarity