Encode/decode proto's map

[tiagopog]

Hey there, @tony612!

First, thank you again for this awesome project :-)

So, I'm trying to work with a mapped field for errors, which is properly defined in the proto file:

message CreateUserResponse {
  bool success = 1;
  string id = 2;
  map<string, string> errors = 3;
}

However it turns out that when I try to use the Elixir's map in the errors key:

Services.CreateUserResponse.new(success: false, errors: %{"foo" => "bar"})

I got this error on the server:

04:41:34.410 [warning] lager_error_logger_h dropped 40 messages in the last second that exceeded the limit of 50 messages/sec
04:41:34.410 [error] CRASH REPORT Process <0.367.0> with 0 neighbours exited with reason:{badkey,'__struct__',#{<<"foo">> => <<"bar">>}} in 'Elixir.Protobuf.Encoder':fix_value/1 line 42 in 'Elixir.GRPC.Adapter.Cowboy.StreamHandler':proc_lib_hack/3 line 135

04:41:34.462 [error] Ranch listener Services.User.Server, connection process #PID<0.365.0>, stream 3 had its request process #PID<0.367.0> exit with reason {:badkey, :__struct__, %{"foo" => "bar"}} and stacktrace [{Protobuf.Encoder, :fix_value, 1, [file: 'lib/exprotobuf/encoder.ex', line: 42]}, {Protobuf.Encoder, :"-fix_undefined/1-fun-0-", 2, [file: 'lib/exprotobuf/encoder.ex', line: 31]}, {Enum, :"-reduce/3-lists^foldl/2-0-", 3, [file: 'lib/enum.ex', line: 1623]}, {Protobuf.Encoder, :encode, 2, [file: 'lib/exprotobuf/encoder.ex', line: 22]}, {GRPC.Server, :stream_send, 2, [file: 'lib/grpc/server.ex', line: 143]}, {GRPC.Adapter.Cowboy.Handler, :init, 2, [file: 'lib/grpc/adapter/cowboy/handler.ex', line: 19]}, {:cowboy_handler, :execute, 2, [file: 'src/cowboy_handler.erl', line: 39]}, {GRPC.Adapter.Cowboy.StreamHandler, :execute, 3, [file: 'lib/grpc/adapter/cowboy/stream_handler.ex', line: 143]}]

04:41:34.463 [error] Ranch listener 'Elixir.Services.User.Server', connection process <0.365.0>, stream 3 had its request process <0.367.0> exit with reason {badkey,'__struct__',#{<<"foo">> => <<"bar">>}} and stacktrace [{'Elixir.Protobuf.Encoder',fix_value,1,[{file,"lib/exprotobuf/encoder.ex"},{line,42}]},{'Elixir.Protobuf.Encoder','-fix_undefined/1-fun-0-',2,[{file,"lib/exprotobuf/encoder.ex"},{line,31}]},{'Elixir.Enum','-reduce/3-lists^foldl/2-0-',3,[{file,"lib/enum.ex"},{line,1623}]},{'Elixir.Protobuf.Encoder',encode,2,[{file,"lib/exprotobuf/encoder.ex"},{line,22}]},{'Elixir.GRPC.Server',stream_send,2,[{file,"lib/grpc/server.ex"},{line,143}]},{'Elixir.GRPC.Adapter.Cowboy.Handler',init,2,[{file,"lib/grpc/adapter/cowboy/handler.ex"},{line,19}]},{cowboy_handler,execute,2,[{file,"src/cowboy_handler.erl"},{line,39}]},{'Elixir.GRPC.Adapter.Cowboy.StreamHandler',execute,3,[{file,"lib/grpc/adapter/cowboy/stream_handler.ex"},{line,143}]}]

After testing a few data structures (simple lists, keyword lists, tuples) I could only make it work with a list of tuples such as:

Services.CreateUserResponse.new(success: false, errors: [{"foo", "bar"}])

Is that the expected behavior? If so, is there any particular reason for choosing to encode/decode proto's maps to this kind of structure (list of tuples) rather than using the Elixir's maps?

Looking forward for your answer. Thanks!

[tony612]

The underlying protobuf implement uses gpb, which use tuples in Erlang for maps in protobuf by default and supports maps as an option as the README shows. I'll try to figure out how to support this in grpc-elixir. Thanks!

[tiagopog]

Thanks for the link, I may dig deeper on it later. It would be really great to have such support for maps in grpc-elixir, I could even try to work on it but first I need to study/understand the project's codebase better :-)

[tiagopog]

Hey there! I took some free time to take a look at both projects exprotobuf and gpc in order to try figuring out a way to bring up this feature to grpc-elixir.

I couldn't make it work yet but so far I've noted that:

1. Protobuf.Parser.parse_files!/2 is called on grpc-elixir with no options being provided, so that it takes the default value which is an empty list.

2. Since the parse functions on exprotobuf end up calling :gpb_parse.post_process_all_files/2, I tried to force manually this list of options to be set with the values which seem to be expected by gpb, I tried [:mapfields_as_maps] and [{:maps, :mapfields_as_maps}]), but none of them worked.

3. Perhaps I'm missing something and the gpb's code comments are not that clear, so I kinda got stuck now. I will inspect the exprotobuf's issues/doc trying to find something useful for this matter and maybe I open new issue there related to this map topic.

Any thoughts on where should I go next?

[tony612]

exprotobuf is replaced by protobuf now. Map has been supported. Doc is coming.

[tiagopog]

That's really awesome to see this feature coming up. I also started to work on an implementation to support maps on exprotobuf but, although it's working, I haven't had time enough to get done with all the stuff related to the feature (tests, doc etc).

I'll try testing what you've done as soon as I get some free time here. Thanks and cheers!