When hex_erl is included as project dependency it tries to run protobuf provider hooks

[kubaodias]

I have a project that includes hex_erl in rebar.config e.g.

{deps,[
    {hex_erl, {git, "https://github.com/hexpm/hex_erl.git", {branch, "master"}}}
]}.

When I run rebar3 compile in my project I get the following error:

===> Verifying dependencies...
===> Fetching hex_erl ({git,"https://github.com/hexpm/hex_erl.git",
                                  {ref,
                                      "6c8973c2ce3acea1ebec8f08d23b4951f272361b"}})
===> Compiling hex_erl
===> Unable to run pre hooks for 'compile', command 'compile' in namespace 'protobuf' not found.

It seems that when protobuf is added as project plugin it isn't fetched as project dependency. https://github.com/hexpm/hex_erl/blob/6c8973c2ce3acea1ebec8f08d23b4951f272361b/rebar.config#L3-L6
https://github.com/hexpm/hex_erl/blob/6c8973c2ce3acea1ebec8f08d23b4951f272361b/rebar.config#L18-L23
Please refer to end of this section in rebar3 documentation: http://www.rebar3.org/v3/docs/using-available-plugins#section-project-plugins-and-overriding-commands

[ferd]

Yeah, project plugins are specifically meant for plugins that override a default command's behavior, and as such are only allowed for top-level apps (i.e. making your own release command). Otherwise just put the plugins in the plugins tuple instead of project_plugins and it should work.

[kubaodias]

@wojtekmach Should hex_erl have a project-only dependency from gpb? Currently I can't inckude hex_erl as dependency of my project without explicitely depending on gpb.

[ericmj]

If I understand correctly rebar3_gpb_plugin should be moved to plugins and rebar3_proper kept in project_plugins. Please let us know if making this change fixes the problem for you.

[ericmj]

Alternatively we can remove protobuf from provider_hooks and manually run the protobuf generator, I think this would be preferable.

[ferd]

rebar3_proper should be a plugin as well.

project_plugins are only to replace a command provided by rebar3, such as release or tar. Anything else (adding a new command) can be a regular plugin. You could move proper to a project plugin to avoid redownloading it, but it ships with no GPL code anymore and wouldn't be that big of a deal

[tsloughter]

No, rebar3_proper should definitely be a project plugin. There is no reason to have it get downloaded when the project is being used as a dependency. That is one of the purposes of project plugins.

As for gpb, unless the generated modules are committed then yes it needs to be a regular plugin.

[wojtekmach]

@tsloughter on master both plugins are project plugins, and yet the keep being downloaded when fetching the package using package as dependency, is there a way to avoid downloads?

See: https://github.com/hexpm/hex_erl/pull/23/files - it's safe to remove provider hooks because we commit generated .erl from .pb. On that branch I switch project_plugins to plugins, btw, which I guess I'll revert, but in either case stuff is being downloaded. Help would be very appreciated.

edit: clarified fetching vs using

[tsloughter]

It is a bug in rebar3 if a project plugin is fetched when the application is used as a dependency of another project.

[wojtekmach]

what's weird is when I add hex_erl on a rebar3 project and do rebar3 compile it does not download gbp, but if I do it on a new mix project it does, even if I switch my ~/.mix/rebar3 version to latest. If someone can reproduce it on mix I'd appreciate, maybe the issue is there after all?

For now, I've removed provider hooks and so part of the issue is solved.

[ferd]

Mix downloads deps its own way, and only gives paths for rebar3 to compile.

[ericmj]

@ferd Mix downloads deps but this is a plugin so it would be rebar3 downloading it when running the bare compiler.

[ericmj]

My guess is that rebar3 bare compiler runs project_plugins because it doesn't know it is being used a dependency. If that's the case the bare compiler should change to behave as a dependency compilation instead or a flag be added that tells it to do so.

[ferd]

ah that could be a possible interpretation. We have no way to know if the bare compiler is used for a dep or a top-level app (another tool than mix could wrap it), and maybe a switch should be needed for that.

[tsloughter]

Yea, that'd be the issue. Not sure if it is easily changed.

[wojtekmach]

Until we fix this in mix and/or rebar3, I've pushed a change that puts gpb stuff into dev profile and protobuf stuff into test profile so that mix will no longer download plugins unnecessarily by default. See: 1e5c558