[Question] Dynamic/Runtime support

[vicb]

Hey Timo,

I have a question for you:

protobufjs supports dynamic code creation to create proto in the runtime from a proto file or a JSON descriptor using protobuf.load().

I wonder:

• if it would be possible to implement this in protobuf-ts,
• if yes how complex it would ?

I can think of 2 ways to do this:

1. Embed the TS compiler and generate ES6 to be eval'd - still we would need a way to parser the proto,
2. Generate the FieldInfo array from the proto file of JSON descriptor. When generating from the proto file there would need to be a runtime proto parser too.

Do you have any thoughts to share about this ?

[timostamm]

Hey Vic,

parsing proto files is the difficult part. There is a EBNF spec, but as far as I am aware, it is not easily possible to generate a parser.

But lets pretend we can parse proto files. I think they should be parsed into FileDescriptors. Protoc parses into FileDescriptors, Plugins work on FileDescriptors. Since they are just protobuf messages, you can easily read/write them in binary or JSON format.

And the Interpreter can already create a MessageType from a FileDescriptor! Lets say you have a FileDescriptor for msg-longs.proto, the code looks like this:

const myFileDesc: FileDescriptor = ...

const registry = DescriptorRegistry.createFrom(myFileDesc);
const interpreter = new Interpreter(registry, ...);

const longsType: IMessageType = interpreter.getMessageType("spec.LongsMessage");

const msg = longsMessageType.create({
  fixed64FieldMinStr: "123"
});

longsType.toJsonString(msg);

There is no generated code at all, it all works with reflection ops.

So a load() function should be pretty easy to provide . Right now, you would have to generate a FileDescriptor using protoc --descriptor_set_out.

Not sure about the benefit, though. It would not be fun to work with messages without the typesafe interface.

Being able to parse a .proto into a FileDescriptor would have an advantage: It would be rather easy to implement an executable that mimics protoc. This means that protobuf-ts would truly be self-hosting, and users would not longer have to install a protoc release at all to compile proto files with a protoc plugin.

[vicb]

parsing proto files is the difficult part

There a probably existing parsers that could be use.
For example mapbox/pbf is based on resolve-protobuf-schema.

I am not sure of what syntax is/is not supported by this project.

Not sure about the benefit, though. It would not be fun to work with messages without the typesafe interface.

The reason why I am asking this question is to understand how easy the migration path from protobuf js would be.
I am using a few google cloud nodejs client that are based on protobufs.

They define both

• a json schema,
• typescript interfaces.

The way it works internally is that they use both the schema (to generate the proto encode/decode code dynamically) and the interface to pass the data.

That is far from ideal but that it the way it is.

Include this library (nodejs-datastore) in my project causes troubles because of a circular dependency in the protobufjs code. It used to work and at some point it broke my rollup build.

Anyway I have asked Google if they would consider moving to protobuf-ts instead as the codebase quality is much better IMO and having support for encoding/decoding proto from a json descriptor (and/or a proto file) would offer them a smoother transition path.

I am not sure if this issue should be left opened as I you have answered all my questions. Thanks !

[timostamm]

Ah, a migration path from protobufjs. Yes, being able to load protobufjs JSON descriptors would certainly make sense.

RPC would need some effort, but loading message types from JSON descriptors at runtime is very much feasible. Let me know in case there is some demand.