Implement remaining unimplemented APIs for Builder types #96805

buyaa-n · 2024-01-10T19:36:03Z

Implemented DefineInitializedDataCore(...), DefineUninitializedDataCore(...), DefinePInvokeMethodCore(...) for TypeBuilerImpl.
Implemented CreateGlobalFunctionsCore(), DefineGlobalMethodCore(...), DefineInitializedDataCore(...), DefineUninitializedDataCore(...), DefinePInvokeMethodCore(...), GetMethodImpl(...), GetMethods() for ModuleBuilerImpl.
requiredCustomModifiers, optionalCustomModifiers for fields, method return type and parameters, which mostly involves Signature changes
Had to refactor type/method/field token generation logic again because of global method/fields
Fix bugs found and reported

Fixes #96436 and #97116
Contributes to #92975

…s, fix bugs found

…ctor field/method/type token logic because of global members

ghost · 2024-01-10T19:36:16Z

Tagging subscribers to this area: @dotnet/area-system-reflection-emit
See info in area-owners.md if you want to be subscribed.

Issue Details

Implemented DefineInitializedDataCore(...), DefineUninitializedDataCore(...), DefinePInvokeMethodCore(...) for TypeBuilerImpl.
Implemented CreateGlobalFunctionsCore(), DefineGlobalMethodCore(...), DefineInitializedDataCore(...), DefineUninitializedDataCore(...), DefinePInvokeMethodCore(...), GetMethodImpl(...), GetMethods() for ModuleBuilerImpl.
requiredCustomModifiers, optionalCustomModifiers for fields, method return type and parameters, which mostly involves Signature changes
Had to refactor type/method/field token generation logic again because of global method/fields
Fix bugs found and reported

Fixes #96436
Contributes to #92975

Author:	buyaa-n
Assignees:	-
Labels:	`area-System.Reflection.Emit`
Milestone:	-

src/libraries/System.Reflection.Emit/src/System/Reflection/Emit/FieldBuilderImpl.cs

...braries/System.Reflection.Emit/src/System/Reflection/Emit/GenericTypeParameterBuilderImpl.cs

src/libraries/System.Reflection.Emit/src/System/Reflection/Emit/MethodBuilderImpl.cs

src/libraries/System.Reflection.Emit/src/System/Reflection/Emit/PseudoCustomAttributesData.cs

src/libraries/System.Reflection.Emit/src/System/Reflection/Emit/TypeBuilderImpl.cs

...es/System.Reflection.Metadata/src/System/Reflection/PortableExecutable/ManagedTextSection.cs

...es/System.Reflection.Emit/tests/PersistableAssemblyBuilder/AssemblySaveModuleBuilderTests.cs

…t/TypeBuilderImpl.cs Co-authored-by: Aaron Robinson <arobins@microsoft.com>

src/libraries/System.Reflection.Emit/src/Resources/Strings.resx

src/libraries/System.Reflection.Emit/src/System/Reflection/Emit/MethodBuilderImpl.cs

src/libraries/System.Reflection.Emit/src/System/Reflection/Emit/SignatureHelper.cs

...ries/System.Reflection.Emit/tests/PersistableAssemblyBuilder/AssemblySaveILGeneratorTests.cs

steveharter · 2024-01-12T15:35:03Z

...ries/System.Reflection.Emit/tests/PersistableAssemblyBuilder/AssemblySaveILGeneratorTests.cs

        public void EmitCalliBlittable()
        {
-            RemoteExecutor.Invoke(() =>


Why was the use of remote executor changed?

Now using collectable ALC instead of default, the discussion is here #96805 (comment)

runtime/src/libraries/System.Reflection.Emit/tests/PersistableAssemblyBuilder/AssemblySaveTools.cs

Lines 12 to 22 in e6d9333

class TestAssemblyLoadContext : AssemblyLoadContext

{

public TestAssemblyLoadContext() : base(isCollectible: true)

{

}

protected override Assembly? Load(AssemblyName name)

{

return null;

}

}

steveharter · 2024-01-12T15:36:37Z

...ries/System.Reflection.Emit/tests/PersistableAssemblyBuilder/AssemblySaveTypeBuilderTests.cs

@@ -515,9 +515,9 @@ public void SaveMultipleGenericTypeParametersToEnsureSortingWorks()
                saveMethod.Invoke(assemblyBuilder, new object[] { file.Path });

                Module m = AssemblySaveTools.LoadAssemblyFromPath(file.Path).Modules.First();
-                Type[] type1Params = m.GetTypes()[2].GetGenericArguments();
+                Type[] type1Params = m.GetTypes()[0].GetGenericArguments();


Why were the items reversed before?

Before the type tokens created on TypeBuilder.CreateType() and before saving the types are ordered with the token order, the types here created in inverse order, you can see that above on row 512-514.

Now with a global type that has global methods/fields this ordering/token assignment not working anymore, now all types and its members tokens created before save (therefore in defined order no matter when TypeBuilder.CreateType() called)

src/libraries/System.Reflection.Emit/src/System/Reflection/Emit/ModuleBuilderImpl.cs

Co-authored-by: Steve Harter <steveharter@users.noreply.github.com>

buyaa-n · 2024-01-12T18:07:21Z

...es/System.Reflection.Emit/tests/PersistableAssemblyBuilder/AssemblySaveModuleBuilderTests.cs

+            Assert.Throws<InvalidOperationException>(() => module.DefineInitializedData("MyField2", new byte[] { 1, 0, 1 }, FieldAttributes.Public));
+        }
+
+        [ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsNotMonoRuntime))]


The Assert at row 257 fails on mono, not sure it is expected ...

Assert.Equal(name + "_0" + "_" + address.ToString(), name + "_" + (address % minAlignmentRequired).ToString() + "_" + address.ToString());

I can see the failure locally after building with mono runtime, though could not find the root cause, so filed an issue.

@buyaa-n I don't understand the test. What is forcing the address to be aligned? is it just based on the length of the blob of data?

I'm looking at sections II.16.3.1 ("Data declaration") and II.16.3.2 ("Accessing data from the PE file") in ECMA-335 and I don't see anything about alignment.

The only mention of alignment I see is:

(Each data item shall have a DataLabel if it is to be referenced afterwards; missing a DataLabel is useful in order to insert alignment padding between data items)

so is DefineInitializedData responsible for adding the padding? And the test is verifying that the right amount got added?

I don't understand the test. What is forcing the address to be aligned? is it just based on the length of the blob of data?

I'm looking at sections II.16.3.1 ("Data declaration") and II.16.3.2 ("Accessing data from the PE file") in ECMA-335 and I don't see anything about alignment.

Right, when I encountered below assert that is failing when a byte array with a size of 1 or 4 (not multiple of 8) is provided for the field RVA data, I also checked ECMA spec and did not found anything about this alignment requirement

runtime/src/libraries/System.Reflection.Metadata/src/System/Reflection/PortableExecutable/ManagedTextSection.cs

Line 190 in 2f63046

Debug.Assert(MappedFieldDataSize % MappedFieldDataAlignment == 0);

It looks like a runtime requirement, not ECMA spec, anyway I had to add alignment for each field RVA:

runtime/src/libraries/System.Reflection.Emit/src/System/Reflection/Emit/ModuleBuilderImpl.cs

Lines 453 to 454 in f36b7cd

fieldDataBuilder.WriteBytes(field._rvaData!);

fieldDataBuilder.Align(ManagedPEBuilder.MappedFieldDataAlignment);

Then this test and the metadata Assert passes.

so is DefineInitializedData responsible for adding the padding? And the test is verifying that the right amount got added?

By my understanding yes, the failing part of the test is asserting if the fields aligned properly. Before adding the alignment same failure was happening locally, now this fail on mono even there is no mono specific code

…ken methods called

...es/System.Reflection.Emit/tests/PersistableAssemblyBuilder/AssemblySaveModuleBuilderTests.cs

buyaa-n · 2024-01-19T21:32:23Z

All failures are unrelated and known, merging

* Implement 'DefinPInvokeMethod', save required/optional CustomModifiers, fix bugs found * Add global method, get method impl and tests * Implement DefineInitializedData(...) and UninitializedData(...), refactor field/method/type token logic because of global members * Update src/libraries/System.Reflection.Emit/src/System/Reflection/Emit/TypeBuilderImpl.cs Co-authored-by: Aaron Robinson <arobins@microsoft.com> * Load assemblies in unloadable context Co-authored-by: Steve Harter <steveharter@users.noreply.github.com> * Add test for DefineUninitializedData(...) * Set parameter count to 0 for RtFieldInfo * Throw when member token is not populated when Module.Get***MetadataToken methods called * Retrieving standalone signature not supported on mono * Create byte array with the size directly instead of null check --------- Co-authored-by: Aaron Robinson <arobins@microsoft.com> Co-authored-by: Steve Harter <steveharter@users.noreply.github.com>

buyaa-n added 3 commits January 7, 2024 22:06

Implement 'DefinPInvokeMethod', save required/optional CustomModifier…

5c467a8

…s, fix bugs found

Add global method, get method impl and tests

f7c67bc

Implement DefineInitializedData(...) and UninitializedData(...), refa…

c09a884

…ctor field/method/type token logic because of global members

dotnet-issue-labeler bot added the area-System.Reflection.Emit label Jan 10, 2024

ghost assigned buyaa-n Jan 10, 2024

buyaa-n requested review from jkotas, steveharter and AaronRobinsonMSFT January 10, 2024 20:37

Merge conflicts

f0bcf1e

AaronRobinsonMSFT reviewed Jan 10, 2024

View reviewed changes

build-analysis bot mentioned this pull request Jan 11, 2024

pip fails to build wheels for cffi (gcc not found) in ubuntu.1804.armarch.open using image alpine-3.15-helix-arm32v7 dotnet/dnceng#1423

Open

3 tasks

Apply feedbacks, fix RVA alignment

1be0fb3

buyaa-n force-pushed the more_impl branch from 02f269d to 1be0fb3 Compare January 11, 2024 19:18

Update src/libraries/System.Reflection.Emit/src/System/Reflection/Emi…

9fbe2cf

…t/TypeBuilderImpl.cs Co-authored-by: Aaron Robinson <arobins@microsoft.com>

buyaa-n added 2 commits January 11, 2024 15:32

Load assemblies in unloadable context

408d7f8

Merge branch 'more_impl' of github.com:buyaa-n/runtime into more_impl

e6d9333

buyaa-n marked this pull request as ready for review January 11, 2024 23:38

buyaa-n requested a review from AaronRobinsonMSFT January 11, 2024 23:39

buyaa-n changed the title ~~Implement remaining unimplemented APIs for ***Builder types~~ Implement remaining unimplemented APIs for Builder types Jan 11, 2024

build-analysis bot mentioned this pull request Jan 12, 2024

CollectionsMarshal.AsSpan assertion failure in System.Text.Json tests #96876

Closed

steveharter reviewed Jan 12, 2024

View reviewed changes

buyaa-n and others added 3 commits January 12, 2024 10:00

Add period to the messages

948447a

Co-authored-by: Steve Harter <steveharter@users.noreply.github.com>

Apply feedback

7164991

Merge branch 'more_impl' of github.com:buyaa-n/runtime into more_impl

4cfa4b9

buyaa-n commented Jan 12, 2024

View reviewed changes

build-analysis bot mentioned this pull request Jan 15, 2024

[browser][MT] GC deadlock #96190

Closed

buyaa-n added 3 commits January 15, 2024 21:07

Add test for DefineUninitializedData(...)

59673e0

Set parameter count to 0 for RtFieldInfo

bb1f5a7

Throw when member token is not populated when Module.Get***MetadataTo…

2627692

…ken methods called

build-analysis bot mentioned this pull request Jan 18, 2024

BasicEventSourceTests.TestsManifestGeneration test failing in CI #97103

Closed

buyaa-n requested a review from steveharter January 18, 2024 05:04

Retreiving standalone signature not supported on mono

f36b7cd

This was referenced Jan 18, 2024

BlobBuilder.Align(int) seems not working on mono #97172

Open

Tracking issue for remaining AssemblyBuilder.Save work in .NET 9 #92975

Closed

steveharter approved these changes Jan 19, 2024

View reviewed changes

...es/System.Reflection.Emit/tests/PersistableAssemblyBuilder/AssemblySaveModuleBuilderTests.cs Outdated Show resolved Hide resolved

Create byte array with the size directly instead of null check

9889bc4

buyaa-n merged commit 62d33ee into dotnet:main Jan 19, 2024
148 of 152 checks passed

buyaa-n deleted the more_impl branch January 19, 2024 21:33

github-actions bot locked and limited conversation to collaborators Feb 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement remaining unimplemented APIs for Builder types #96805

Implement remaining unimplemented APIs for Builder types #96805

buyaa-n commented Jan 10, 2024 •

edited

ghost commented Jan 10, 2024

steveharter Jan 12, 2024

buyaa-n Jan 12, 2024

steveharter Jan 12, 2024

buyaa-n Jan 12, 2024

buyaa-n Jan 12, 2024

buyaa-n Jan 18, 2024

lambdageek Jan 18, 2024

buyaa-n Jan 18, 2024

buyaa-n commented Jan 19, 2024

	class TestAssemblyLoadContext : AssemblyLoadContext
	{
	public TestAssemblyLoadContext() : base(isCollectible: true)
	{
	}

	protected override Assembly? Load(AssemblyName name)
	{
	return null;
	}
	}

	fieldDataBuilder.WriteBytes(field._rvaData!);
	fieldDataBuilder.Align(ManagedPEBuilder.MappedFieldDataAlignment);

Implement remaining unimplemented APIs for Builder types #96805

Implement remaining unimplemented APIs for Builder types #96805

Conversation

buyaa-n commented Jan 10, 2024 • edited

ghost commented Jan 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

buyaa-n commented Jan 19, 2024

buyaa-n commented Jan 10, 2024 •

edited