Reflection support

[Aniket21mathur]

One key feature provided by protocol message classes is reflection. You can iterate over the fields of a message and manipulate their values without writing your code against any specific message type. One very useful way to use reflection is for converting protocol messages to and from other encodings, such as XML or JSON. A more advanced use of reflection might be to find differences between two messages of the same type, or to develop a sort of "regular expressions for protocol messages" in which you can write expressions that match certain message contents.

[Aniket21mathur]

@mppf commented ->

About reflection - these are interesting. I'm not sure it's critical that we implement these right away. Do you see any reason we would need to change the existing code or API in a significant way if we add them? (Other than adding methods and types).
It says:
You can iterate over the fields of a message and manipulate their values without writing your code against any specific message type.
And yet the example does use a specific message type (it says new Foo). Is it possible, in the C++ version, to read a message from an input stream without knowing what message it is, and then use these reflection mechanisms?
I think the question here is - would it be sufficient to allow people to write generic code for many message types (we can already do that in Chapel using the Reflection module). Or, do we need to add a dynamic mechanism similar to what is provided here?
Note that since we have used a record for the message types (which I think is appropriate) - it is not possible to arrange for the equivalent of Message* foo = new Foo; in the Chapel code. This line is creating a Foo which is a subclass of Message and then it is forgetting the Foo part of the type (at compile-time at least). This is a thing one can do with classes but not with records in Chapel. In the future we hope to add a dynamic type supporting interface s / constrained generics that would allow it, though.

[Aniket21mathur]

@mppf

Do you see any reason we would need to change the existing code or API in a significant way if we add them?

I don't think it will require a significant change in the existing APIs. We will have to add an extra interface over the existing APIs. That interface will provide complete information about the messages in the proto files and we will be able to write more generic code.

An example use case of reflection APIs in C#.

Message descriptors (the information in the .proto file) and instances of messages can be examined programmatically using the reflection API. This can be useful when writing generic code such as a different text format or a smart diff tool. Each generated class has a static Descriptor property, and the descriptor for any instance can be retrieved using the IMessage.Descriptor property. As a quick example of how these can be used, here is a short method to print the top-level fields of any message.

public void PrintMessage(IMessage message)
{
    var descriptor = message.Descriptor;
    foreach (var field in descriptor.Fields.InDeclarationOrder())
    {
        Console.WriteLine(
            "Field {0} ({1}): {2}",
            field.FieldNumber,
            field.Name,
            field.Accessor.GetValue(message);
    }
}

We can pass any message type to the above generic function and operate on its fields. We can also modify the above code to set field values of the message and serialize it, the converse is also true.

The C# proto plugin dumps the entire proto message as a base64 encoded string in the generated file. Please see https://github.com/Aniket21mathur/ProtocolBuffers-Examples/blob/master/c%23/build/FriendRequest.cs#L15. This is then parsed by the language side library to generate message descriptor objects with field accessors and other useful functions.

And yet the example does use a specific message type (it says new Foo). Is it possible, in the C++ version, to read a message from an input stream without knowing what message it is, and then use these reflection mechanisms?

I think the C++ example creates a bit of confusion, the C# one is more to the point. I don't think it should be possible to read a message from an input stream without knowing the message type, the decoding algorithms should not allow it(As far as I know).

would it be sufficient to allow people to write generic code for many message types (we can already do that in Chapel using the Reflection module). Or, do we need to add a dynamic mechanism similar to what is provided here?

I guess it would be good to start with the generic code, something similar to the above C# example. Able to write generic code is the basic use of reflection that I see. We can move to more advanced usage latter if we find it useful :)

[mppf]

Re. the PrintMessage example from C#, we can already do this sort of thing:

import Reflection;

record myRecord {
  var a: int;
  var b: real;
}

var exampleRecord = new myRecord(1, 200.0);

for param i in 0..<Reflection.numFields(myRecord) {
  writeln("Field number=", i,
          " name=", Reflection.getFieldName(myRecord, i),
          " value=", Reflection.getField(exampleRecord, i));
}

If protobuf users need some other specific "field number" then we might need a particular method to translate, but I think the Reflection module already covers most of this.

[Aniket21mathur]

Yes, the reflection module do a lot of job for us. I see we can also set field values of records with the reflection module.

If protobuf users need some other specific "field number" then we might need a particular method to translate, but I think the Reflection module already covers most of this.

Yes, we should have something which returns the actual field number of the proto field.