EnC: metadata rows added by a delta are invisible to lookups once the table's lookup hash is built (ApplyUpdate fails: 'The assembly update failed')

[dr1rrb]

Summary

CMiniMdRW::GenericFindWithHash builds a lazy per-table lookup hash once a table exceeds INDEX_ROW_COUNT_THRESHOLD (25) rows, and from then on searches only the hash — there is no linear fallback. The EnC delta-apply path (CMiniMdRW::ApplyDelta → ApplyTableDelta) appends rows without maintaining or invalidating m_pLookUpHashes. As a result, any row an EnC delta adds after the hash was built is invisible to metadata lookups.

For the FieldRVA table this is fatal to hot reload: Roslyn (with the new-in-.NET-10 AddFieldRva capability, #112446) emits one FieldRVA row per generation whenever an edited method body contains an array initializer or a u8 literal (a fresh <PrivateImplementationDetails> per generation). During EditAndContinueModule::ApplyEditAndContinue, the runtime looks the new row up via GetFieldRVA; once the aggregate FieldRVA table has crossed 25 rows and the hash was built, the lookup returns CLDB_E_RECORD_NOTFOUND and the whole update fails with:

System.InvalidOperationException: The assembly update failed.
   at System.Reflection.Metadata.MetadataUpdater.ApplyUpdate(...)

Every subsequent update of the session then fails the same way (the compiler-side baseline has moved on). With a baseline of B FieldRVA rows, hot reload dies at edit max(2, 25 − B + 1) — real-world apps easily have dozens of FieldRVA rows, so this kills hot reload after one or two edits.

We hit this in Uno Platform's hot-reload CI (unoplatform/uno#23863): the dev-server EnC session died on the 4th edit of any XAML file (that app's baseline has 22 FieldRVA rows: 25 − 22 + 1 = 4; every generated .g.cs contains array initializers), and the poisoned session then failed every later update until the 60-minute job timeout.

The same staleness affects every GenericFindWithHash-backed lookup on tables grown by EnC deltas (Constant, FieldMarshal, ClassLayout, FieldLayout, ImplMap, FieldRVA, NestedClass) — FieldRVA is the one on the hot apply path that hard-fails the update.

Reproduction

Self-contained, single Program.cs (below): builds a tiny library whose baseline has 26 FieldRVA rows (25 distinct u8 literals + one array initializer), loads it, starts a Roslyn 5.6 EnC session with the .NET 10 capability set (including AddFieldRva), then applies three method-body edits, each containing an array initializer.

$ DOTNET_MODIFIABLE_ASSEMBLIES=debug dotnet run -c Release

MetadataUpdater.IsSupported = True
Baseline FieldRVA rows      = 26
Predicted failing update    = 2 (lookup-hash threshold 25)
baseline M() => gen0-last=0
gen1: applied, M() => gen1-last=1
gen2: APPLY FAILED — InvalidOperationException: The assembly update failed.

Generation 1 applies (the hash is built during that very lookup, so it still contains the row); generation 2's row is appended after the hash exists and is invisible. Also reproducible by replaying captured deltas with a bare Assembly.LoadFrom + MetadataUpdater.ApplyUpdate loop — no Roslyn on the apply side.

FieldRvaEncRepro.csproj

<Project Sdk="Microsoft.NET.Sdk">
	<PropertyGroup>
		<OutputType>Exe</OutputType>
		<TargetFramework>net10.0</TargetFramework>
		<ImplicitUsings>enable</ImplicitUsings>
	</PropertyGroup>
	<ItemGroup>
		<!-- Roslyn is only used to EMIT the EnC deltas, like any hot-reload host does; the bug
		     itself is in the runtime's MetadataUpdater.ApplyUpdate metadata merging. -->
		<PackageReference Include="Microsoft.CodeAnalysis.CSharp.Workspaces" Version="5.6.0" />
		<PackageReference Include="Microsoft.CodeAnalysis.Features" Version="5.6.0" />
		<PackageReference Include="Microsoft.CodeAnalysis.CSharp.Features" Version="5.6.0" />
	</ItemGroup>
</Project>

Program.cs

using System.Collections.Immutable;
using System.Reflection;
using System.Reflection.Metadata;
using System.Reflection.PortableExecutable;
using System.Reflection.Metadata.Ecma335;
using System.Runtime.CompilerServices;
using System.Runtime.Loader;
using System.Text;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.Text;

// ---------------------------------------------------------------------------------------
// 1. Baseline: a class padded with 25 distinct u8 literals (25 FieldRVA rows) + a method
//    whose body contains an array initializer we edit at every generation.
// ---------------------------------------------------------------------------------------
string SourceFor(int gen)
{
	var sb = new StringBuilder();
	sb.AppendLine("using System;");
	sb.AppendLine("public static class C");
	sb.AppendLine("{");
	for (var i = 0; i < 25; i++)
	{
		// 25 distinct blobs => 25 <PrivateImplementationDetails> data fields with FieldRVA.
		sb.AppendLine($"\tpublic static ReadOnlySpan<byte> D{i} => \"padding-data-{i:D2}-abcdefghijklmnopqrstuvwxyz\"u8;");
	}
	sb.AppendLine($"\tpublic static string M()");
	sb.AppendLine("\t{");
	sb.AppendLine($"\t\tvar data = new byte[] {{ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, {gen} }};");
	sb.AppendLine($"\t\treturn $\"gen{gen}-last={{data[15]}}\";");
	sb.AppendLine("\t}");
	sb.AppendLine("}");
	return sb.ToString();
}

var work = Directory.CreateTempSubdirectory("fieldrva-enc-repro").FullName;
var sourcePath = Path.Combine(work, "lib.cs");
var dllPath = Path.Combine(work, "lib.dll");
var pdbPath = Path.Combine(work, "lib.pdb");
var utf8NoBom = new UTF8Encoding(encoderShouldEmitUTF8Identifier: false);
var text0 = SourceText.From(SourceFor(0), utf8NoBom);
File.WriteAllBytes(sourcePath, utf8NoBom.GetBytes(SourceFor(0)));

var refPack = Directory.GetFiles(Path.Combine(
	Path.GetDirectoryName(typeof(object).Assembly.Location)!.Replace("shared/Microsoft.NETCore.App", "packs/Microsoft.NETCore.App.Ref"), "ref", "net10.0"), "*.dll");
var references = refPack.Select(p => (MetadataReference)MetadataReference.CreateFromFile(p)).ToArray();
var parseOptions = CSharpParseOptions.Default;
var compilationOptions = new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary, optimizationLevel: OptimizationLevel.Debug);

var tree = CSharpSyntaxTree.ParseText(text0, parseOptions, path: sourcePath);
var compilation = CSharpCompilation.Create("lib", [tree], references, compilationOptions);
var emitOptions = new Microsoft.CodeAnalysis.Emit.EmitOptions(debugInformationFormat: Microsoft.CodeAnalysis.Emit.DebugInformationFormat.PortablePdb);
using (var peStream = File.Create(dllPath))
using (var pdbStream = File.Create(pdbPath))
{
	var emit = compilation.Emit(peStream, pdbStream, options: emitOptions);
	if (!emit.Success)
	{
		Console.WriteLine(string.Join(Environment.NewLine, emit.Diagnostics));
		return 3;
	}
}

int baselineFieldRvaRows;
using (var pe = new PEReader(File.OpenRead(dllPath)))
{
	baselineFieldRvaRows = pe.GetMetadataReader().GetTableRowCount(TableIndex.FieldRva);
}

var asm = new AssemblyLoadContext("target").LoadFromAssemblyPath(dllPath);
Console.WriteLine($"MetadataUpdater.IsSupported = {MetadataUpdater.IsSupported}");
Console.WriteLine($"Baseline FieldRVA rows      = {baselineFieldRvaRows}");
Console.WriteLine($"Predicted failing update    = {Math.Max(2, 25 - baselineFieldRvaRows + 1)} (lookup-hash threshold 25)");
Console.WriteLine($"baseline M() => {asm.GetType("C")!.GetMethod("M")!.Invoke(null, null)}");

// ---------------------------------------------------------------------------------------
// 2. EnC session over the baseline. UnitTestingHotReloadService is the stable internal
//    entry point hot-reload hosts use; reflection because it is internal — the bug being
//    demonstrated is on the APPLY side, not here.
// ---------------------------------------------------------------------------------------
var hostAssemblies = Microsoft.CodeAnalysis.Host.Mef.MefHostServices.DefaultAssemblies
	.Add(Assembly.Load("Microsoft.CodeAnalysis.Features"))
	.Add(Assembly.Load("Microsoft.CodeAnalysis.CSharp.Features"));
var workspace = new AdhocWorkspace(Microsoft.CodeAnalysis.Host.Mef.MefHostServices.Create(hostAssemblies));
var projectId = ProjectId.CreateNewId();
var documentId = DocumentId.CreateNewId(projectId);
var projectInfo = ProjectInfo.Create(projectId, VersionStamp.Create(), "lib", "lib", LanguageNames.CSharp)
	.WithFilePath(Path.Combine(work, "lib.csproj"))
	.WithParseOptions(parseOptions)
	.WithCompilationOptions(compilationOptions)
	.WithMetadataReferences(references)
	.WithCompilationOutputInfo(default(CompilationOutputInfo).WithAssemblyPath(dllPath))
	.WithDocuments([DocumentInfo.Create(documentId, "lib.cs", loader: TextLoader.From(TextAndVersion.Create(text0, VersionStamp.Create(), sourcePath)), filePath: sourcePath)]);
var solution = workspace.CurrentSolution.AddProject(projectInfo);

var features = Assembly.Load("Microsoft.CodeAnalysis.Features");
var serviceType = features.GetType("Microsoft.CodeAnalysis.ExternalAccess.UnitTesting.Api.UnitTestingHotReloadService")!;
var service = Activator.CreateInstance(serviceType, workspace.Services)!;

// The capability set reported by the .NET 10 CoreCLR — AddFieldRva included, which makes
// Roslyn emit the RVA-based data-field form in deltas (one FieldRVA row per generation).
var capabilities = "Baseline AddMethodToExistingType AddStaticFieldToExistingType AddInstanceFieldToExistingType NewTypeDefinition ChangeCustomAttributes UpdateParameters GenericUpdateMethod GenericAddMethodToExistingType GenericAddFieldToExistingType AddFieldRva"
	.Split(' ').ToImmutableArray();

await (Task)serviceType.GetMethod("StartSessionAsync", [typeof(Solution), typeof(ImmutableArray<string>), typeof(CancellationToken)])!
	.Invoke(service, [solution, capabilities, CancellationToken.None])!;
var emitMethod = serviceType.GetMethod("EmitSolutionUpdateAsync", [typeof(Solution), typeof(bool), typeof(CancellationToken)])!;

for (var gen = 1; gen <= 3; gen++)
{
	solution = solution.WithDocumentText(documentId, SourceText.From(SourceFor(gen), utf8NoBom));

	var task = (Task)emitMethod.Invoke(service, [solution, true, CancellationToken.None])!;
	await task;
	var tuple = (ITuple)task.GetType().GetProperty("Result")!.GetValue(task)!;
	var diagnostics = (ImmutableArray<Diagnostic>)tuple[1]!;
	if (diagnostics.Any(d => d.Severity == DiagnosticSeverity.Error))
	{
		Console.WriteLine($"gen{gen}: EMIT BLOCKED — {string.Join(" | ", diagnostics)}");
		return 2;
	}

	foreach (var update in (System.Collections.IEnumerable)tuple[0]!)
	{
		var t = update.GetType();
		var md = (ImmutableArray<byte>)t.GetField("MetadataDelta")!.GetValue(update)!;
		var il = (ImmutableArray<byte>)t.GetField("ILDelta")!.GetValue(update)!;
		var pdb = (ImmutableArray<byte>)t.GetField("PdbDelta")!.GetValue(update)!;
		try
		{
			MetadataUpdater.ApplyUpdate(asm, md.AsSpan(), il.AsSpan(), pdb.AsSpan());
		}
		catch (Exception e)
		{
			Console.WriteLine($"gen{gen}: APPLY FAILED — {e.GetType().Name}: {e.Message}");
			return 1;
		}
	}

	Console.WriteLine($"gen{gen}: applied, M() => {asm.GetType("C")!.GetMethod("M")!.Invoke(null, null)}");
}

Console.WriteLine("ALL 3 GENERATIONS APPLIED — bug not reproduced (fixed runtime?)");
return 0;

Root cause analysis (traced on a Checked CoreCLR, v10.0.10)

• src/coreclr/md/enc/metamodelrw.cpp — INDEX_ROW_COUNT_THRESHOLD is 25; CMiniMdRW::GenericFindWithHash searches only the hash when it exists (no fallback); GenericBuildHashTable faults the hash in at the first lookup once ridEnd > INDEX_ROW_COUNT_THRESHOLD - 1.
• src/coreclr/md/enc/metamodelenc.cpp — CMiniMdRW::ApplyTableDelta (added-record branch) appends via AddRecord/Add*Record and never touches m_pLookUpHashes. (Non-EnC insertion paths — e.g. AddCustomAttribute in the emitter — do maintain their dedicated hashes; the EnC apply path does not.)
• src/coreclr/vm/encee.cpp — EditAndContinueModule::ApplyEditAndContinue → GetFieldRVA(token) → FindFieldRVAHelper → GenericFindWithHash (the FieldRVA table is unsorted after EnC additions) → not found → CLDB_E_RECORD_NOTFOUND bubbles out as the generic InvalidOperationException.

Trace from a Checked build with logging added to FindFieldRVAHelper (four identical XAML edits of the Uno test app, baseline B = 22):

gen1: FindFieldRVAHelper HASH tk=04000d81 -> rid=23 (cRecs=23)   APPLY OK
gen2: FindFieldRVAHelper HASH tk=04000d82 -> rid=24 (cRecs=24)   APPLY OK
gen3: FindFieldRVAHelper HASH tk=04000d83 -> rid=25 (cRecs=25)   APPLY OK   <- hash built during this lookup
gen4: FindFieldRVAHelper HASH tk=04000d84 -> rid=0  (cRecs=26)   APPLY FAIL <- row exists, hash misses it

Candidate fix (validated)

Invalidating the table's lookup hash when the EnC apply adds a record makes the identical deltas apply (verified on the same Checked build — the previously failing 4-delta chain then applies 4/4). PR incoming.

Regression?

The metadata defect is long-standing, but it was unreachable through hot reload before .NET 10: pre-AddFieldRva Roslyn never emitted FieldRVA delta rows (array initializers used the element-wise EnC fallback codegen).

Host-side workaround until fixed: withdraw AddFieldRva from the capability set passed to the EnC session — Roslyn then falls back to the pre-.NET-10 element-wise array-initializer codegen (no FieldRVA delta rows, identical semantics). This is what Uno's dev-server ships in the meantime.

Configuration

• .NET SDK 10.0.302, Microsoft.NETCore.App 10.0.10, linux-x64 (also reproduced on win-x64 CI agents)
• Roslyn 5.6.0 for delta emission (any AddFieldRva-aware emitter applies)

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @steveisok, @tommcdon, @dotnet/dotnet-diag
See info in area-owners.md if you want to be subscribed.