Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.Reflection.Emit.AssemblyBuilder.Save #4491

Open
jonorossi opened this issue Nov 13, 2015 · 23 comments

Comments

Projects
None yet
@jonorossi
Copy link
Contributor

commented Nov 13, 2015

AssemblyBuilder.Save and AssemblyBuilderAccess.RunAndSave isn't available in .NET Core, however coreclr seems to have the code to implement it but I looks conditionally compiled out.

https://github.com/dotnet/coreclr/blob/bc146608854d1db9cdbcc0b08029a87754e12b49/src/mscorlib/src/System/Reflection/Emit/AssemblyBuilder.cs#L1690-L1711

Our use case in Castle DynamicProxy is to write out dynamically created assemblies to disk so we can run peverify over the assembly in unit tests. It also greatly helps writing out the assembly and opening it in ildasm to manually verify IL.

Is there any chance we can get this functionality that .NET Framework has introduced into .NET Core, or an alternative way of doing the same thing. This might not sound like a big deal if you aren't familiar with Castle DynamicProxy's internals, but it really is as it heavily reduces our confidence in both our code and the .NET runtime as well as reducing our ability to track down defects.

@joshfree

This comment has been minimized.

Copy link
Member

commented Nov 13, 2015

Related issue dotnet/coreclr#1709

@bartdesmet

This comment has been minimized.

Copy link
Contributor

commented Jan 6, 2016

Another interesting omission is the lack of LambdaExpression.Compile accepting a MethodBuilder which tends to be very useful when combined with AssemblyBuilder.Save support.

@jkotas

This comment has been minimized.

Copy link
Member

commented Feb 28, 2016

Large chunks of Reflection.Emit code in the runtime are Windows-specific. It is not easy to just enable them for CoreCLR because of it would not work on Unix. In particular, the support for emitting debug information depends on unmanaged PDB writer that is very complex Windows-specific component that we have no plans to open source and bring x-plat.

We are also not happy about Reflection.Emit being integrated deeply into the runtime. Reflection.Emit should be a independent higher level component that generates PE file that the runtime loads as any other assembly (note that runtime supports loading assemblies from byte array, so the PE file does not have to be persisted on disk). IKVM.Reflection.Emit done by Jeroen Frijters proved that it is pretty doable and can actually work much better than the current Reflection.Emit built into the runtime.

The plan to address the Reflection.Emit is:

  1. (Done) Create fast standalone managed metadata, IL and portable PDB writer that can be used by Roslyn, and anybody else to produce managed assemblies. @tmat has been working on it: dotnet/roslyn#7683.
  2. Build Reflection.Emit object model as a layer on top of the standalone managed metadata writer to make it easier for existing projects that use Reflection.Emit to migrate to .NET Core.
@jonorossi

This comment has been minimized.

Copy link
Contributor Author

commented Feb 29, 2016

@jkotas many thanks for the explanation here, it makes complete sense. I'm glad you guys are planning to provide a more general purpose object model around the heart of .NET, IL. I'd seen the portable PDB reader/writer and some metadata object model appear in Roslyn, they look great. In the past I've used Cecil for the exact reason the .NET Framework API is so heavily tied into the runtime, glad to see more stuff being pulled out of the runtime.

@tmat

This comment has been minimized.

Copy link
Member

commented Feb 29, 2016

@karelz

This comment has been minimized.

Copy link
Member

commented Feb 28, 2017

Next step: We need API proposal with deep analysis how implementable it is. The old Desktop API might be an option, likely not a good one.
This is quite hard to do and requires lots of experience.

@KevinRansom

This comment has been minimized.

Copy link
Member

commented May 30, 2017

@karelz @jkotas --- What is the status of this feature? When can we expect to see dynamic codegen with persistence on the coreclr?

@karelz

This comment has been minimized.

Copy link
Member

commented May 30, 2017

There is currently no plan (milestone Future + my comment above) - area owners @AtsushiKan @DnlHarvey @joshfree can provide more details.

@stakx

This comment has been minimized.

Copy link
Collaborator

commented Jan 6, 2018

@AtsushiKan @DnlHarvey @joshfree, one question about the plan that @jkotas laid out above:

We are also not happy about Reflection.Emit being integrated deeply into the runtime. Reflection.Emit should be a independent higher level component that generates PE file that the runtime loads as any other assembly [...].

The plan to address the Reflection.Emit is:

  1. [...]
  2. Build Reflection.Emit object model as a layer on top of the standalone managed metadata writer to make it easier for existing projects that use Reflection.Emit to migrate to .NET Core.

I do not fully understand the plan, in particular with regard to dynamic assemblies (AssemblyBuilder.DefineDynamicAssembly) and their ability to be incrementally updated / augmented with new dynamic types. How would this still be possible if assemblies must first be fully written out as a byte stream or file, then be loaded as a whole?

I imagine one would have to write each dynamically generated type to a single assembly... but wouldn't this in turn interfere with internal type visibility and type member accessibility between generated types that should all be in the same assembly, and with stuff like [assembly: InternalsVisibleTo("DynamicallyGeneratedAssembly") (on which e. g. DynamicProxy currently depends to be able to proxy internal user types)?

Would it be possible to load more than one assembly having the exact same name (whether weak or strong), so that each such single-type assembly can have a [assembly: InternalsVisibleTo] attribute for its own name, such that types in the separate assemblies can reference each other's internals?

@jkotas

This comment has been minimized.

Copy link
Member

commented Jan 6, 2018

their ability to be incrementally updated / augmented with new dynamic types

Yes, there would need to be a runtime hook for this.

Conceptually, each type addition in Reflection.Emit produces a new assembly that contains all existing metadata (with all existing tokens preserved) plus the new type. Then the runtime is asked to update the metadata of the already loaded assembly to the new version.

Sending the whole assembly over for each edit is not efficient for large assemblies with small edits. It would be better to just send the metadata delta with the new stuff over. Fortunately, this exists in the runtime for edit and continue already. Look for https://github.com/dotnet/coreclr/search?q=ApplyEditAndContinue.

So exposing the ApplyDelta metadata API for the standalone Reflection.Emit implementation should do the trick here.

@masonwheeler

This comment has been minimized.

Copy link

commented May 7, 2018

Large chunks of Reflection.Emit code in the runtime are Windows-specific. It is not easy to just enable them for CoreCLR because of it would not work on Unix. In particular, the support for emitting debug information depends on unmanaged PDB writer that is very complex Windows-specific component that we have no plans to open source and bring x-plat.

The existence of cross-platform Roslyn suggests that this isn't a particularly intractable problem, at a high level at least.

Could this please be made a high priority? The inability to use Reflection.Emit is currently my #1 blocker preventing me from migrating existing work to Core, and as such it's far more urgent, from my perspective at least, than the development of new features.

@zgramana

This comment has been minimized.

Copy link

commented May 7, 2018

Here's a good starting point for a non-Windows implementation (and it's rather mature): https://github.com/mono/mono/tree/master/mcs/class/corlib/System.Reflection.Emit

@stakx

This comment has been minimized.

Copy link
Collaborator

commented May 11, 2018

@AtsushiKan: Out of curiosity, is there any documentation / spec at all about the data format used for EnC metadata deltas, other than the actual CoreCLR source code?

@ghost

This comment has been minimized.

Copy link

commented May 11, 2018

Out of curiosity, is there any documentation / spec at all about the data format used for EnC metadata deltas, other than the actual CoreCLR source code?

Not that I've heard of but EnC is not an area where I have any knowledge.

@stakx

This comment has been minimized.

Copy link
Collaborator

commented May 11, 2018

@AtsushiKan - Thanks for the quick reply. 👍 I almost expected this to be a really arcane area, but I thought I'd give it a try and ask. I'll see what I can gather together on my own.

@karelz

This comment has been minimized.

Copy link
Member

commented May 11, 2018

@stakx EnC MetaData format was never thoroughly documented (neither publicly nor internally). It was always defined just in the implementation.

@tmat

This comment has been minimized.

Copy link
Member

commented May 11, 2018

@stakx Curious, what do you need it for?

@stakx

This comment has been minimized.

Copy link
Collaborator

commented May 11, 2018

@karelz - I almost suspected so... Thanks for this info! 👍

@tmat - Mostly out of curiousity.

I have spent a lot of time with System.Reflection in the past few months, and noticed its various limitations or bugs. There are sometimes situations where you'd rather just see plain, uninterpreted metadata instead of the semi-high-level view that Reflection provides—which is neither original metadata, nor as high-level as e.g. C#, but something of its own sitting somewhere in-between.

I have also discovered System.Reflection.Metadata (SRM). I have been thinking about how nice it would be if the (Core)CLR had the necessary runtime hooks and extension points so that both Reflection and Reflection.Emit could be rebuilt on top of SRM, outside of mscorlib / System.Private.CoreLib. (At the same time, this would also enable folks to build their own custom reflection / dynamic code emission libraries.) Two things appear to be missing as of today to make this possible:

  • Neither the CLR nor the CoreCLR runtimes allow user code to get at the metadata stream & IL of loaded assemblies/modules, so it could be inspected using e.g. SRM. (This assumes that the metadata as per ECMA-335 is still available unaltered after a module has been loaded by the CLR, which likely isn't the case.)

  • Neither the CLR nor the CoreCLR runtime allows altering/updating an assembly that's been loaded via Assembly.Load(byte[]) (which is what you can use today to load an assembly generated with SRM). Perhaps if EnC was exposed in some way, updating a loaded assembly would become possible.

I realise that having such runtime hooks closely allied with System.Reflection.Metadata is probably a pipe dream at this stage. But it might allow some super-interesting scenarios. So I'm trying to learn how this part of the runtime works to better gauge how big of a project this would be.

If there are any plans or efforts in this direction, I'd love to learn about it & be directed to the appropriate places. Otherwise... I don't intend to hijack this issue, which is just about AssemblyBuilder.Save. 😉

@jkotas

This comment has been minimized.

Copy link
Member

commented May 11, 2018

Neither the CLR nor the CoreCLR runtimes allow user code to get at the metadata stream & IL of loaded assemblies/modules, so it could be inspected using e.g. SRM

Have you seen https://github.com/dotnet/coreclr/blob/85374ceaed177f71472cc4c23c69daf7402e5048/src/System.Private.CoreLib/src/System/Reflection/Metadata/AssemblyExtensions.cs#L25 ?

@stakx

This comment has been minimized.

Copy link
Collaborator

commented May 11, 2018

@jkotas - Yes, I have, and that's a really nice extension point! But it comes with some limitations: It's a Try... method and so may fail, e.g. for dynamically generated assemblies (AssemblyBuilder); something that today's Reflection (IIRC) can deal with just fine. So it seems Reflection couldn't be built on top of that to work as reliably as it does today. But I was definitely thinking of something like this. 👍

@vermorel

This comment has been minimized.

Copy link

commented Mar 3, 2019

We have finally rolled our own open-source version of AssemblyBuilder.Save based on System.Reflection.Metadata (and no further dependencies). It targets .NET Standard and works under .NET Core. It does support dynamically generated assemblies.

Check-out https://github.com/Lokad/ILPack Feedback will be most welcome.

@masonwheeler

This comment has been minimized.

Copy link

commented Mar 12, 2019

@vermorel Looks interesting. Does it generate PDBs too?

@vermorel

This comment has been minimized.

Copy link

commented Mar 12, 2019

@masonwheeler ILPack does not generate PDB.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.