System.Reflection.Metadata should provide easier signature reading API

[nguerrera]

EDIT 12/3/2015 - Replaced entire description with the detailed proposal matching PR dotnet/corefx#4809

Today, System.Reflection.Metadata provides low-level access to ECMA-335 CLI metadata, but only provides signatures as blobs that must be parsed with direct knowledge of the format as described in the section II.23.2 of the CLI specification. There is a BlobReader for reading various elements out of blobs, but it is up to the caller to read the right things at the right positions.

This was by-design as the library is designed to sit at the lowest level behind the scenes of higher level API such as Reflection proper, Roslyn, or CCI.

The challenge with signatures is that they are variable-length and encode tree structures, and each higher level model that could sit on top of System.Reflection.Metadata will have different representations for the trees. We do not want to introduce an API to build a fully-formed tree that then has to be traversed and rewritten to match the actual use case.

This proposal is therefore a middle ground between:

1. Here are the bytes and some helpers. Read the spec to decode. (Status quo)
2. Yet another high-level metadata API. (Out-of-scope for this layer)

It works as follows:

1. The caller chooses an arbitrary representation, TType, for type symbols and implements ISignatureTypeProvider<TType>. (See full API spec below).
2. Signatures are parsed by recursive descent and the provider is called to create new type nodes:

• Give me the TType that represents this primitive
• Give me the TType that represents this TypeDefinition
• Give me the TType that represents an array of this other TType
• etc.

Sample Usage

Given a suitable TypeSymbol and TypeSymbolProvider : ISignatureTypeProvider<TypeSymbol>, here is code walking all of the TypeSymbol's for every field, parameter, and return type:

using (var stream = File.OpenRead(pathToDll))
using (var peReader = new PEReader(stream))
{
    var reader = peReader.GetMetadataReader();
    var provider = new TypeSymbolProvider();

    foreach (TypeDefinitionHandle typeHandle in reader.TypeDefinitions)
    {
         TypeDefinition type = reader.GetTypeDefinition(typeHandle);

         foreach (FieldDefinitionHandle fieldHandle in type.GetFields())
         {
              FieldDefinition field = reader.GetFieldDefinition(fieldHandle);
              TypeSymbol fieldType = field.DecodeSignature(provider);
              // ...
         }

         foreach (MethodDefinitionHandle methodHandle in type.GetMethods())
         {
              MethodDefinition method = reader.GetMethodDefinition(methodHandle);
              MethodSignature<TypeSymbol> methodSig = method.DecodeSignature(provider);
              TypeSymbol returnType = methodSig.ReturnType;
              // ...

              foreach (TypeSymbol parameterType in methodSig.ParameterTypes)
              {
                  // ...
              }
         }
    }
}

Full API

Additions to existing types

These provide convenience entry points. There are other use cases where you want to parse only part of a signature or a signature that you did not obtain from the metadata reader. For that, the SignatureDecoder

namespace System.Reflection.Metadata 
{
    public struct FieldDefinition
    {
        public TType DecodeSignature<TType>(
            ISignatureTypeProvider<TType> provider, 
            SignatureDecoderOptions options=SignatureDecoderOptions.None);
    }

    public struct MemberReference
    {
        public TType DecodeFieldSignature<TType>(
            ISignatureTypeProvider<TType> provider, 
            SignatureDecoderOptions options=SignatureDecoderOptions.None);

        public MethodSignature<TType> DecodeMethodSignature<TType>(
            ISignatureTypeProvider<TType> provider, 
            SignatureDecoderOptions options=SignatureDecoderOptions.None);

         // MemberReferenceKind GetKind(); already exists
    }

    public struct MethodDefinition
    {
        public MethodSignature<TType> DecodeSignature<TType>(
            ISignatureTypeProvider<TType> provider,
            SignatureDecoderOptions options=SignatureDecoderOptions.None);
    }

    public struct MethodSpecification
    {
        public ImmutableArray<TType> DecodeSignature<TType>(
            ISignatureTypeProvider<TType> provider,
            SignatureDecoderOptions options=SignatureDecoderOptions.None);
    }

    public struct PropertyDefinition
    {
        public MethodSignature<TType> DecodeSignature<TType>(
            ISignatureTypeProvider<TType> provider,
            SignatureDecoderOptions options=SignatureDecoderOptions.None);
    }

    public struct StandaloneSignature
    {
        public ImmutableArray<TType> DecodeLocalSignature<TType>(
            ISignatureTypeProvider<TType> provider,
            SignatureDecoderOptions options=SignatureDecoderOptions.None);

        public MethodSignature<TType> DecodeMethodSignature<TType>(
            ISignatureTypeProvider<TType> provider, 
            SignatureDecoderOptions options=SignatureDecoderOptions.None);

        public StandaloneSignatureKind GetKind();
    }

    public enum StandaloneSignatureKind
    {
        LocalVariables = 1,
        Method = 0,
    }

    public struct TypeSpecification
    {
        public TType DecodeSignature<TType>(
            ISignatureTypeProvider<TType> provider, 
            SignatureDecoderOptions options=SignatureDecoderOptions.None);
    }
}

New types

namespace System.Reflection.Metadata.Decoding
{
    public struct ArrayShape
    {
        public ArrayShape(int rank, ImmutableArray<int> sizes, ImmutableArray<int> lowerBounds);
        public ImmutableArray<int> LowerBounds { get; }
        public int Rank { get; }
        public ImmutableArray<int> Sizes { get; }
    }

    public interface IConstructedTypeProvider<TType> : ISZArrayTypeProvider<TType>
    {
        TType GetArrayType(TType elementType, ArrayShape shape);
        TType GetByReferenceType(TType elementType);
        TType GetGenericInstance(TType genericType, ImmutableArray<TType> typeArguments);
        TType GetPointerType(TType elementType);
    }

    public interface IPrimitiveTypeProvider<TType>
    {
        TType GetPrimitiveType(PrimitiveTypeCode typeCode);
    }

    public interface ITypeProvider<TType>
    {
        TType GetTypeFromDefinition(
            MetadataReader reader, 
            TypeDefinitionHandle handle, 
            SignatureTypeHandleCode code);
        TType GetTypeFromReference(
            MetadataReader reader, 
            TypeReferenceHandle handle, 
            SignatureTypeHandleCode code);
    }

    public interface ISignatureTypeProvider<TType> 
        : IConstructedTypeProvider<TType>, 
          IPrimitiveTypeProvider<TType>, 
          ITypeProvider<TType>
    {
        TType GetFunctionPointerType(MethodSignature<TType> signature);
        TType GetGenericMethodParameter(int index);
        TType GetGenericTypeParameter(int index);
        TType GetModifiedType(
          MetadataReader reader, 
          bool isRequired, 
          EntityHandle modifierTypeHandle, 
          TType unmodifiedType);
        TType GetPinnedType(TType elementType);
    }

    public interface ISZArrayTypeProvider<TType>
    {
        TType GetSZArrayType(TType elementType);
    }

    public struct MethodSignature<TType>
    {
        public MethodSignature(
            SignatureHeader header,
            TType returnType,
            int requiredParameterCount,
            int genericParameterCount,
            ImmutableArray<TType> parameterTypes);

        public int GenericParameterCount { get; }
        public SignatureHeader Header { get; }
        public ImmutableArray<TType> ParameterTypes { get; }
        public int RequiredParameterCount { get; }
        public TType ReturnType { get; }
    }

    public enum PrimitiveTypeCode : byte
    {
        Boolean = (byte)2,
        Byte = (byte)5,
        Char = (byte)3,
        Double = (byte)13,
        Int16 = (byte)6,
        Int32 = (byte)8,
        Int64 = (byte)10,
        IntPtr = (byte)24,
        Object = (byte)28,
        SByte = (byte)4,
        Single = (byte)12,
        String = (byte)14,
        TypedReference = (byte)22,
        UInt16 = (byte)7,
        UInt32 = (byte)9,
        UInt64 = (byte)11,
        UIntPtr = (byte)25,
        Void = (byte)1,
    }

    public struct SignatureDecoder<TType>
    {
        public SignatureDecoder(
            ISignatureTypeProvider<TType> provider, 
            MetadataReader metadataReader=null, 
            SignatureDecoderOptions options=SignatureDecoderOptions.None);

        public TType DecodeFieldSignature(ref BlobReader blobReader);
        public ImmutableArray<TType> DecodeLocalSignature(ref BlobReader blobReader);
        public MethodSignature<TType> DecodeMethodSignature(ref BlobReader blobReader);
        public ImmutableArray<TType> DecodeMethodSpecificationSignature(ref BlobReader blobReader);
        public TType DecodeType(ref BlobReader blobReader);
    }

    public enum SignatureDecoderOptions
    {
        DifferentiateClassAndValueTypes = 1,
        None = 0,
    }

    public enum SignatureTypeHandleCode : byte 
    {
        Class = (byte)18,
        Unresolved = (byte)0,
        ValueType = (byte)17,
    }
 }

Notes

• The interface segregation of ISignatureTypeProvider<T> is for future TypeNameParser and CustomAttributeDecoder (still under development in dev/metadata branch), which share some but not all of the same requirements for a type provider as the SignatureDecoder

[sharwell]

For reference, here's the classes I've developed for my own use.

tunnelvisionlabs/dotnet-compatibility#24

I like the fact that they are lightweight, but they are probably sub-optimal for certain use cases for a few reasons, such as lack of caching nested signatures.

[nguerrera]

Early draft of signature decoder is in dev/metadata branch now (has been for a while) but I didn't get around to updating this issue. Priority is lower on this as we won't ship it until some time after VS 2015 RTM. Any help refining the API and/or adding tests is welcome in the meantime.

[devhawk]

Can you publish the mini spec?

[nguerrera]

@devhawk done.

[nguerrera]

@terrajobst This is ready for API review.

[nguerrera]

Design changes to discuss:

1. Make pinned a property of local and not of TType. Change DecodeLocalSignature to return array of (pinned, type).
2. Move PrimitiveTypeCode up one level or get rid of it in favor of individual GetXxx() methods
3. Merge IPrimitiveTypeProvider into ITypeProvider.

[terrajobst]

We reviewed the API today and have some minor feedback. Please switch back to api-ready-for-review when addressed.

[tmat]

@nguerrera I think we implemented all changes we wanted. Can we close now?

[nguerrera]

Yes, all of the changes were implemented.

[devhawk]

👍

[sharwell]

@nguerrera Can you update the Milestone for this issue?

Edit: I just realized it might not be possible since this is just one assembly in a repository with many.