Snake case support, by_proto prefix, oneof/maps fix

[tsloughter]

This PR covers the issues: #73 #74 and #75

In addition to adding msg_name_to_snake_case, by_proto and the fixes for oneof/maps this also has a few changes to make it work better with rebar3. Mainly moving the compile_descriptor hook to a pre_hook instead of post.

[tomas-abrahamsson]

Thanks, will take a look

[tomas-abrahamsson]

I don't think moving the compile_descriptor from a post-hook to a pre-hook is good.

It is a two-stage process to build gpb (or three-stage if you include building the gpb_version.hrl):

1. First build the compiler
2. Then use the compiler to build an encoder for .proto files. This is the descriptor. It can be used to add a protobuf-encoded description of the proto file(s) into generated .erl files, if the descriptor' (-descr') option is specified.

On this branch, make clean followed by rebar compile now leads to:

% make clean
rm -f include/gpb_version.hrl ...
...
% rebar compile
==> gpb (compile)
Compiling descriptor.proto...
{"init terminating in do_boot",{undef,[{gpb_compile,c,[['descriptor.proto']],[]},{init,start_it,1,[]},{init,start_em,1,[]}]}}

Maybe there's some other way to make this work with both rebar2 and rebar3? (not knowing what the problem with rebar3 was)

[tsloughter]

Aah. Well damn.. hm. So with rebar3 it does not compile to ./ebin but to _build/default/lib/gpb/ebin. With the use of pre the hook would write to ./ebin/ which gets copied to _build/default/lib/gpb/ebin when things are compiled.

I guess I'll have to add a check to have it compile to _build if it exists and ./ebin if it exists.

[tomas-abrahamsson]

Ah, I see, didn't think of it compiling to _build/ even though I've seen it

[tomas-abrahamsson]

Maybe it could be worth adding also unit tests for the oneof/map crash in #75, for example the below (this is what I used locally to verify the change)

can_tolower_record_names_with_oneof_test() ->
    {ok, Defs} = parse_lines(["message Msg1 {",
                              "  oneof u {",
                              "    Msg1   a = 1;",
                              "    uint32 b = 2;",
                              "  }",
                              "}"]),
    [{{msg,msg1}, [#gpb_oneof{fields=[#?gpb_field{name=a,type={msg,msg1}},
                                      #?gpb_field{name=b}]}]}] =
        do_process_sort_defs(Defs, [msg_name_to_lower]).

can_tolower_record_names_with_map_test() ->
    {ok, Defs} = parse_lines(["message Msg1 {",
                              "  required uint32 f = 1;",
                              "}"
                              "message Msg2 {"
                              "  map<string,Msg1> m = 1;",
                              "}"]),
    [{{msg,msg1}, [#?gpb_field{}]},
     {{msg,msg2}, [#?gpb_field{type={map,string,{msg,msg1}}}]}] =
        do_process_sort_defs(Defs, [msg_name_to_lower]).

[tomas-abrahamsson]

hmm... just remembered I hadn't taken a look at this part. Sorry for doing it late. And also sorry in advance for bringing up a naming issue. :)

The prefix naming adds the option to specify prefix for individual messages on a per-message basis. Earlier, we discussed prefixing on a per-proto-file basis. I'm fine with either way, I think, but now that it is on a per-message basis, I see two things:

1. The by_proto name should be something else. by_msg comes to mind, but I'm of course open to other alternatives
2. The key that one should specify for prefixing messages is the name after any conversions (to-lower/snake etc) but since this is far from obvious, I think the by_msg (or whatever) sub-option would deserve a line or two in the (oh-so-long) doc text for gpb_compile:file.

[tsloughter]

Hah, I keep thinking "messages" are "protos" and not "messages in a proto". I'll look at making it actually by_proto. Probably have that tomorrow.

[tsloughter]

Hm, post_process_all_files has all msgs, including imported, already in the list of defs and no way (as far as I can tell) to know which proto each message came from, so I don't think that is going to work.

[tomas-abrahamsson]

Should add that I've run my usual battery of tests on a local test-merge (of just-refetched b6e67e5), and from that point of view, things look good.

[tsloughter]

I updated it to be by_msg but am happy to change it back to by_proto with a good way to distinguish which proto each message came from.

[tomas-abrahamsson]

... happy to change it back to by_proto with a good way to distinguish which proto each message came from.

Yes, one would have to keep track of which messages belong to which files, roughly like this:

1. in post_process_one_file (or in the calling code, where the file name is known) add a tuple to the list, let's say {{msg_containment,ProtoFileName},MsgNamesForThisProtoFile}
2. in post_process_all_files, one would have see the concatenation of all such lists of defs, including several entries of {{msg_containment,...},[...]} (one such entry per seen/imported proto file) and do prefixing according to this.
3. The msg_containment tuples could either be left in the list (but then the message names ought to be reformatted there too, so the view is coherent) or else the msg_containment tuples could be removed since they're not used any more.

[tsloughter]

Ok, this sounds workable, taking a crack at it now.

[tsloughter]

Updated back to by_proto.

[tomas-abrahamsson]

Line is overlong

[tomas-abrahamsson]

Looks generally fine, but one eunit test is failing, the fix should be trivial, I should think. The rest of the tests pass.

in function gpb_compile_tests:parses_file_to_msg_defs_test/0 (test/gpb_compile_tests.erl, line 180)
**error:{badmatch,{ok,[{{msg,'Msg'},[{field,field1,1,2,uint32,required,[]}]},
               {{msg_containment,"X"},['Msg']}]}}
  output:<<"">>

After this, I'd like to suggest squashing some commits, and then I'd be ready to merge it.

[tsloughter]

Updated and squashed to 2 commits.

[tomas-abrahamsson]

Thanks! Merged and included in just-pushed 3.25.0 (closing)