Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

when imported proto file changed, rebar3 compile will not recompile the proto, any ideas? #30

Closed
zhangr011 opened this issue Oct 12, 2016 · 29 comments

Comments

@zhangr011
Copy link

---- rebar.config ----
{provider_hooks, 
 [
  {pre, [{compile, {protobuf, compile}},
         {clean,   {protobuf, clean}}]}
 ]}.
---- pb_17.proto ----
syntax = "proto3";

package protobuf;
import "pb_common.proto";

...
---- pb_common.proto ----
syntax = "proto3";

package protobuf;

...

when pb_common.proto changed, rebar3 compile will not compile pb_17.proto.

@tomas-abrahamsson
Copy link
Contributor

One way to go to find all imports could be to use the (recently added) import_fetcher option (or something similarly named, not at a computer at the moment, typing from memory) and perhaps the to_proto_defs option to just run the parser stage, not the code generator.

@zhangr011
Copy link
Author

Sorry, i dont know how to do it yet. When check gpb_compile:file(), I don't find any issue about import_fetcher.

@lrascao
Copy link
Owner

lrascao commented Oct 12, 2016

@zhangr011 is pb_common.proto in the same dir as pb_17.proto?
Also could you post the output of running DEBUG=1 rebar3 compile please?

@zhangr011
Copy link
Author

yes, they are in the same dir.
this is for debug info

===> making sure that target erl dir "/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/src/protocol" exists
===> making sure that target hrl dir "/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/include" exists
===> reading proto files from ["./subgit/p05_proto"], generating "_pb.erl" to "/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/src/protocol" and "_pb.hrl" to "/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/include"
===> compiling "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_common.proto"
===> opts: [{i,"./subgit/p05_proto"},
                   {module_name_suffix,"_pb"},
                {o_erl,"/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/src/protocol"},
                   {o_hrl,"/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/include"},
                   {strings_as_binaries,false},
                   type_specs,
                   {i,"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto"},
                   {i,"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto"}]
===>      Compiled pb_common.proto
===>      Skipped pb_9.proto
===>      Skipped pb_50.proto
===>      Skipped pb_40.proto
===>      Skipped pb_30.proto
===>      Skipped pb_21.proto
===>      Skipped pb_19.proto
===>      Skipped pb_18.proto
===>      Skipped pb_17.proto
===>      Skipped pb_16.proto
===>      Skipped pb_15.proto
===>      Skipped pb_14.proto
===>      Skipped pb_13.proto
===>      Skipped pb_12.proto
===>      Skipped pb_11.proto
===>      Skipped pb_10.proto

when pb_common.proto changed, the proto with "import pb_commom.proto" has been skipped.

@lrascao
Copy link
Owner

lrascao commented Oct 20, 2016

i'm using the import_fetcher functionality, soon i'll have something working

@tomas-abrahamsson
Copy link
Contributor

I was toying in gpb with some functions gpb_compile:locate_import and gpb_compile:read_import, which might be of use for this case. I just pushed them to github to a branch, locate-import. If you'd need it, I can push it to master, or make any changes if necessary to make it more useful. Just let me know

@lrascao
Copy link
Owner

lrascao commented Oct 20, 2016

@tomas-abrahamsson any advice on extracting a list of imports from a .proto file? i got it working with the process dictionary however it feels a bit dirty...

@tomas-abrahamsson
Copy link
Contributor

Hmm... there's always gpb_parse:fetch_imports(Defs). Maybe that's the way to go then, {ok, Defs, _Warnings} = gpb_compile:file(..., [..., to_proto_defs, return]) followed by [gpb_parse:locate_import(Imp) || Imp <- lists:usort(gpb_parse:fetch_imports(Defs))] and let that be the list of files to check. Maybe with error handling if a proto file contains errors so gpb:compile:file returns error instead. (lists:usort because a file could possibly be imported several times)

@lrascao
Copy link
Owner

lrascao commented Oct 21, 2016

much better, thanks!
the gpb_compile:locate_import proved very useful, would you mind merging and tagging a new release with that?

@tomas-abrahamsson
Copy link
Contributor

Perfect, 3.26.4 is now out containing this. (had to make a small adjustment to a type spec, so it is not exactly the commits on the previous branch, but rebased ones)

@lrascao
Copy link
Owner

lrascao commented Oct 21, 2016

@zhangr011 could you please try out the latest develop branch? if everything looks good i'll cut a new release

@zhangr011
Copy link
Author

@lrascao thanks. i get the plugin from rebar3 plugin, i don't know how to update it yet. i'll check it later, sorry.

@lrascao
Copy link
Owner

lrascao commented Oct 27, 2016

when you get the time it's just a matter of cloning the plugin repo and checking out develop, then on your project you create a _checkouts folder and create a symlink to the plugin repo

@zhangr011
Copy link
Author

zhangr011 commented Oct 29, 2016

@lrascao

- rebar.config -
{plugins, [rebar3_gpb_plugin]}.
_checkouts path for branch develop:
├── _checkouts
│   └── rebar3_gpb_plugin -> /Users/zhangrong/repository/rebar3_gpb_plugin

when rebar3 compile, got error:

===> Package gpb-3.26.4 not found. Fetching registry updates and trying again...
===> Updating package registry...
===> Plugin rebar3_gpb_plugin not available. It will not be used.
===> Command compil not found

sorry, i dont't know how to make _checkout work yet, can u help me?

ps: same error when i check out gpb into _checkouts

├── _checkouts
│   ├── gpb -> /Users/zhangrong/repository/gpb/
│   └── rebar3_gpb_plugin -> /Users/zhangrong/repository/rebar3_gpb_plugin

@zhangr011
Copy link
Author

@lrascao
update ok, but when pb_commom.proto changed, rebar3 compile get this error.

===> proto files found: ["/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_common.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_9.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_50.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_40.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_30.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_21.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_19.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_18.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_17.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_16.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_15.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_14.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_13.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_12.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_11.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_10.proto"]
===> compiling "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_common.proto" to "/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/src/protocol/pb_common_pb.erl"
===> opts: [{i,"./subgit/p05_proto"},
                   {module_name_suffix,"_pb"},
                   {o_erl,"/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/src/protocol"},
                   {o_hrl,"/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/include"},
                   {strings_as_binaries,false},
                   type_specs,
                   {i,"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto"},
                   {i,"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto"}]
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: undef
===> Stack trace to the error location:
[{gpb_compile,locate_import,
              ["pb_common.proto",
               [{i,"./subgit/p05_proto"},
                {module_name_suffix,"_pb"},
                {o_erl,"/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/src/protocol"},
                {o_hrl,"/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/include"},
                {strings_as_binaries,false},
                type_specs,
                {i,"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto"},
                {i,"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto"}]],
              []},
 {rebar3_gpb_compiler,'-filter_included_proto/3-fun-0-',2,
                      [{file,"/Users/zhangrong/repository/leshu/server_p05/_checkouts/rebar3_gpb_plugin/src/rebar3_gpb_compiler.erl"},
                       {line,131}]},
 {lists,map,2,[{file,"lists.erl"},{line,1239}]},
 {rebar3_gpb_compiler,filter_included_proto,3,
                      [{file,"/Users/zhangrong/repository/leshu/server_p05/_checkouts/rebar3_gpb_plugin/src/rebar3_gpb_compiler.erl"},
                       {line,130}]},
 {lists,filtermap,2,[{file,"lists.erl"},{line,1317}]},
 {rebar3_gpb_compiler,compile,4,
                      [{file,"/Users/zhangrong/repository/leshu/server_p05/_checkouts/rebar3_gpb_plugin/src/rebar3_gpb_compiler.erl"},
                       {line,103}]},
 {rebar3_gpb_compiler,compile,1,
                      [{file,"/Users/zhangrong/repository/leshu/server_p05/_checkouts/rebar3_gpb_plugin/src/rebar3_gpb_compiler.erl"},
                       {line,49}]},
 {lists,foreach,2,[{file,"lists.erl"},{line,1338}]}]
===> When submitting a bug report, please include the output of `rebar3 report "your command"`

rebar3 compile again, only got pb_common.proto generate too.

@lrascao
Copy link
Owner

lrascao commented Oct 29, 2016

the undef error on gpb_compile:locate_import seems to indicate you're not on gpb 3.26.4, can you try deleting _checkouts/gpb and making a fresh build?

@zhangr011
Copy link
Author

zhangr011 commented Oct 29, 2016

@lrascao seems not work yet, only pb_common.proto be regenerated, maybe _checkout is not work ...

zhangrongdeMacBook-Pro:server_p05 zhangrong$ git status
On branch dev_combat
Your branch is ahead of 'origin/dev_combat' by 1 commit.
  (use "git push" to publish your local commits)
Changes not staged for commit:
  (use "git add ..." to update what will be committed)
  (use "git checkout -- ..." to discard changes in working directory)
    modified:   include/pb_common_pb.hrl
    modified:   src/protocol/pb_common_pb.erl
Untracked files:
  (use "git add ..." to include in what will be committed)
    Error
    _checkouts/
    p05_build.tar.gz
    sh.zip
no changes added to commit (use "git add" and/or "git commit -a")
zhangrongdeMacBook-Pro:server_p05 zhangrong$ 
zhangrongdeMacBook-Pro:server_p05 zhangrong$ 
zhangrongdeMacBook-Pro:server_p05 zhangrong$ ls -l _checkouts/
total 16
lrwxr-xr-x  1 zhangrong  staff  32 Oct 29 11:33 gpb -> /Users/zhangrong/repository/gpb/
lrwxr-xr-x  1 zhangrong  staff  45 Oct 29 10:53 rebar3_gpb_plugin -> /Users/zhangrong/repository/rebar3_gpb_plugin

@lrascao
Copy link
Owner

lrascao commented Oct 29, 2016

i meant removing _checkouts/gpb (rm _checkouts/gpb)
and cleaning the build: rm -rf _build

@zhangr011
Copy link
Author

@lrascao it seems not work for me.

  1. when i rm -rf _build/default/plugins, with _checkouts/rebar3_gpb_plugin, not work

    zhangrongdeMacBook-Pro:server_p05 zhangrong$ rebar3 compile
    ===> Verifying dependencies...
    ===> Compiling server
    ===> Unable to run pre hooks for 'compile', command 'compile' in namespace 'protobuf' not found.
    
  2. when i ln -s rebar3_gpb_plugin_develop _build/default/plugins, then like this:

    ls -l _build/default/plugins/
    total 16
    lrwxr-xr-x  1 zhangrong  staff  32 Oct 31 11:06 gpb -> /Users/zhangrong/repository/gpb/
    lrwxr-xr-x  1 zhangrong  staff  46 Oct 31 11:05 rebar3_gpb_plugin -> /Users/zhangrong/repository/rebar3_gpb_plugin/
    

    seem's work, rebar3_gpb_plugin get log

    commit 49ba32d3a9d67590310395c2a7cfad8a7919fbec
    Author: Luis Rascao 
    Date:   Fri Oct 21 01:07:24 2016 +0100
    Refactor compilation strategy
    Drop rebar's base compiler and manually check
    dependencies to compile, make use of module
    prefix and suffix, also correctly build protos that
    include other protos when the dependency is change.
    
  3. but same as before, when pb_common.proto changed, only pb_common.proto compiled.

    Changes not staged for commit:
    (use "git add ..." to update what will be committed)
    (use "git checkout -- ..." to discard changes in working directory)
      modified:   include/pb_common_pb.hrl
      modified:   src/protocol/pb_common_pb.erl
    

    gpb version is 3.26.4 now

    diff --git a/include/pb_common_pb.hrl b/include/pb_common_pb.hrl
    index 7dae9fe..c8d837a 100644
    --- a/include/pb_common_pb.hrl
    +++ b/include/pb_common_pb.hrl
    @@ -1,11 +1,11 @@
    %% -*- coding: utf-8 -*-
    %% Automatically generated, do not edit
    -%% Generated by gpb_compile version 3.24.4
    +%% Generated by gpb_compile version 3.26.4
    

@lrascao
Copy link
Owner

lrascao commented Oct 31, 2016

Can you post the output of 'DEBUG=1 rebar3 compile'?

@zhangr011
Copy link
Author

===> Compiling server
===> making sure that target erl dir "/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/src/protocol" exists
===> making sure that target hrl dir "/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/include" exists
===> reading proto files from ["./subgit/p05_proto"], generating ".erl" to "/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/src/protocol" and ".hrl" to "/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/include"
===> proto files found: ["/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_common.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_9.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_50.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_40.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_30.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_21.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_19.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_18.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_17.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_16.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_15.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_14.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_13.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_12.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_11.proto",
                                "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_10.proto"]
===> compiling "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_common.proto" to "/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/src/protocol/pb_common_pb.erl"
===> opts: [{i,"./subgit/p05_proto"},
                   {module_name_suffix,"_pb"},
                   {o_erl,"/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/src/protocol"},
                   {o_hrl,"/Users/zhangrong/repository/leshu/server_p05/_build/default/lib/server/include"},
                   {strings_as_binaries,false},
                   type_specs,
                   {i,"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto"},
                   {i,"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto"}]
===> protos that include "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_common.proto": []
===> run_hooks("/Users/zhangrong/repository/leshu/server_p05", pre_hooks, compile) -> no hooks defined
===> run_hooks("/Users/zhangrong/repository/leshu/server_p05", pre_hooks, erlc_compile) -> no hooks defined
===> erlopts [debug_info,bin_opt_info,debug_info,fail_on_warning,
                     {i,"./_build/default/plugins/gpb/include"},
                     {lager_print_records_flag,false},
                     {parse_transform,lager_transform}]

@zhangr011
Copy link
Author

===> protos that include "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_common.proto": []

is [] ...

@lrascao
Copy link
Owner

lrascao commented Oct 31, 2016

weird, is this a public project? can i clone it somehow to debug the problem?

@zhangr011
Copy link
Author

yes, i can send you the test unit by email, u can check it.
some issue may help.
the two files are same package, due to gpb cant identify pb_common.pbId32 as different package.

syntax = "proto2";
package protobuf;
import "pb_common.proto";
...
message pbError{
        optional int32   error_code = 1;
        repeated string  args = 2;
        optional pbId32   value  = 3;  %% not pb_common.pbId32
}

@lrascao
Copy link
Owner

lrascao commented Nov 1, 2016

i think i've found the issue, could you please try the latest commit in develop (f15f354)?

@zhangr011
Copy link
Author

goods news, got include files. bad things, only recompiled, not re generated.

===> protos that include "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_common.proto": ["/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_9.proto",
"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_50.proto",
"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_40.proto",
"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_30.proto",
"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_19.proto",
"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_18.proto",
"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_17.proto",
"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_16.proto",
"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_15.proto",
"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_13.proto",
"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_12.proto",                                                                                                                  "/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_11.proto",
"/Users/zhangrong/repository/leshu/server_p05/subgit/p05_proto/pb_10.proto"]
Changes not staged for commit:
  (use "git add ..." to update what will be committed)
  (use "git checkout -- ..." to discard changes in working directory)
    modified:   include/pb_common_pb.hrl
    modified:   src/protocol/pb_common_pb.erl

@lrascao
Copy link
Owner

lrascao commented Nov 2, 2016

found that one as well, please try again latest develop (d9ddfb5)

@zhangr011
Copy link
Author

it works like a charm. thank you very much.

@lrascao
Copy link
Owner

lrascao commented Nov 4, 2016

1.10.0 has just been cut with this fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants