Add support for non-Erlang/OTP (raw) dependencies

[alavrik]

It would be nice to be able to use rebar for dealing with non-Erlang dependencies
such as C libraries, build scripts, protocol definitions, etc. Here is my suggestion:

Introduce a new 'raw' option for dependency specs in rebar.config file.
For example:

{deps,
    {dependency_name, "1.0.*",
     {git, "<...>", {branch, "master"}},
     [raw]
    }
]}.

When this option is specified, rebar does not require the dependency to
have a standard Erlang/OTP layout which assumes presence of either
"src/dependency_name.app.src" or "ebin/dependency_name.app" files.

[alavrik]

@Tuncer, @dizzyd, what do you think? I rely on this feature myself, and a few other people contacted me asking if it could be contributed to upstream.

[alavrik]

Here are some details about the problem I'm trying to solve.

We have a bunch of Erlang projects that have various non-Erlang dependencies. Some of non-Erlang stuff is used in the build process, some need to be included into generated releases, etc (more details below). The dependencies live in svn and git repositories.

Naturally, as I'm already using rebar for managing dependencies and building Erlang projects, I would like to use it for the remaining non-Erlang bits as well. Moreover, for us, those non-Erlang dependencies are tightly integrated in the dependency graphs and not much different from regular Erlang deps.

My initial approach was to make such non-Erlang dependencies appear as if they were Erlang apps. In other words, I was just adding fake src/*.app.src file to their top-level directories. This kind of works, but it doesn't look very nice and not universally applicable, say, to third-party projects to which I'm not able to add src/*.app.src file.

The solution I'm proposing is to mark such dependencies as non-Erlang in rebar.config in the first place. This way, rebar won't expect src/*.app.src (or ebin/*.app) files and won't check for their existence.

This new feature is really simple. It doesn't do anything automatically beyond downloading dependencies. Essentially, you use rebar to download raw dependencies and then use a combination of hooks to work with them by locating their directories through the exported REBAR_DEPS_DIR shell variable.

Some more details:

In our applications, we rely on protocol and data structure definitions. Some protocol definitions are shared among projects and all of them are used only at build time. Protocol definitions reside in their own git repositories and independently versioned. For exampe:

protocol_defs1
protocol_defs2
protocol_defs3: protocol_defs1 protocol_defs2

Erlang app1: protocol_defs1

Erlag app2: protocol_defs2

Erlang app3: protocol_defs3

Erlang node1: app1 app2 app3

Another example -- build support scripts. We have a common project where such scripts live (in fact, we have several of them for specialized needs). We pull such repositories in as non-Erlang dependencies and then use hooks to call scripts during compilation.

In addition to that, we have a set of third-party C libraries and executables pre-built for our target deployment platforms. Such libs and binaries are used by Erlang applications and included into generated releases. They live in svn repositories because git is not good at efficiently dealing with big binary files as git repositories grow out of control when new versions are added.

[hyperthunk]

You can do this without touching rebar's source code:

{post_hooks, [
    {'get-deps', "git clone -b blah https://github.com/blah/blah.git deps/blah"}
]}.

[alavrik]

@hyperthunk, it wouldn't be a good solution for us. This is not a one-off problem. We have a growing set of non-Erlang dependencies for Erlang projects. Some of them are in git, others in svn. They are really first class dependencies and I'd like to be able to use full set of rebar -deps commands to work with them (i.e. not only get-deps, but also update-deps and list-deps).

[hyperthunk]

I'm reminded of a conversation on the Haskell mailing list about how cabal is not a package manager. But instead of doing this, perhaps rebar should only enforce the OTP check when it's actually trying to build something. Then you could use the alien plugin. Also, there's nothing to stop you from hooking all the -deps commands.

[alavrik]

Hooking -deps commands is not an option when you have 20+ developers, 10s of projects and several vcs. Yes, this can be automated outside of rebar, but why do this if there an easy way to add this functionality as a small and backward-compatible native rebar feature? Wouldn't it be useful to have this feature in rebar?

[hyperthunk]

Listen man, I've wanted this feature myself and have written plugins to have to deal with it which were much more complex because they had to circumvent rebar's deps features altogether. So you're preaching to the choir while I'm playing devil's advocate! :)

I'm not a maintainer so ultimately it's not up to me, but at least you've already given some compelling reasons for the patch by the time they come to consider it.

[hyperthunk]

Btw in what way does the number of developers have anything to do with this discussion? But I've often wanted to deal with native code dependencies and resource bundles (common configuration and so on) so I do understand the intent.

[alavrik]

@hyperthunk The number of users and projects is an inverse measure of how much automation or documentation you need for common tasks in order to prevent errors and wasted time. For one or two projects, defining hooks for -deps and copy-pasting them would probably work fine (of course, the current solution of adding a fake src/X.app.src files is a lot simpler). However, with many active projects and complicated dependency graphs, following such practice wouldn't be a good idea.

rebar itself is a good example of the right level of automation. Instead of writing and maintaining shell scripts and Makefiles for getting dependencies and driving compilation, you provide one simple small config file and the tool does all the necessary magic.

[dstar4138]

A maintainer want to chime in? I saw your comment on stackoverflow and I actually like this syntax better, adding options to the dependencies is a good idea that can be built upon in later versions of rebar (such as disabling features from included dependencies like turning off port drivers or not including dependencies wrapped in releases). Just my two cents...

[dizzyd]

Reading it over today...

[alavrik]

@dizzyd any updates? Is there anything I can do to speed up the decision/merge?

[essen]

I would like this too. It would be very useful for Farwest.

[hyperthunk]

After thinking over this carefully, I can not only see the point but it would make trying to rebar-ize RabbitMQ possible rather than just a nice little day dream of mine.

+1

[ghost]

We have to carefully define what it will mean for the other commands
and how it will be ignored. This could mean not adding raw deps to the
list of directories to pre-process.

[alavrik]

@Tuncer, based on my experience with using this, I don't need rebar to interpret compile or eunit for non-Erlang dependencies. I'm happy to use hooks to do exactly what I need once the dependencies are downloaded by rebar. In fact, I would be opposed to the idea of rebar trying to guess what to do with my non-Erlang stuff.

get-deps, update-deps, check-deps and list-deps should work the same way they behave with regular Erlang dependencies, except that they should not check the version correspondence. This is why I needed this feature in the first place.

As far as I can tell, all of this should be covered by the change in the pull request. It is very simple and usable "as is" with no extra fitting required. For instance, both compile and eunit commands do nothing when the dependency doesn't look like an Erlang/OTP application, i.e. when it doesn't have ebin/*.app or src/*.app.src file.

[miebach]

+1 for merging this. It seems very straight forward and clean to have raw dependencies. Using hooks is not really the same.

[hyperthunk]

+1 for the feature, but now that @Tuncer has merged in the no-state branch that allows configuration to be altered during {pre|post}process/2 and {pre|post}(command-name) functions it is possible to abstract this behaviour into a plugin without affecting rebar_deps and the commands listed update-deps, check-deps, list-deps can all still continue to work.

It may require some extra exports in rebar_deps though. I'm definitely not arguing against this (or playing devil's advocate) any more - just questioning whether it should be an optional plugin or a core baked-in feature.

[dizzyd]

Let's say, hypothetically, that we merged this functionality. Can you provide concrete examples of how you would use hooks, etc. to use these deps?

[hyperthunk]

@dizzyd is that question directed at me?

[dizzyd]

Question was directed at @alavrik who initially submitted the patch.

[alavrik]

@dizzyd we use use raw deps for downloading build tools and protocol definitions. We generate Erlang code from protocol definitions as a rebar compile step.

Build tools are called from Makefiles, which are in turn called from rebar.config pre-compile hooks. For example:

{pre_hooks, [
    {compile, "make -C src -f Makefile.piqi"},
    ...

{post_hooks, [
    {clean, "make -C src -f Makefile.piqi clean"},
    ...

Inside those Makefiles, we rely on the REBAR_DEPS_DIR variable to locate necessary build scripts, e.g. PIQIC = $(REBAR_DEPS_DIR)/piqi/priv/bin/piqic-erlang.

Some raw dependencies with build scripts are downloaded as rebar dependencies, but they are never used from rebar. Instead, build procedures, including os packaging, are driven by separate makefiles.

Raw deps with protocol definitions are used in several ways. Sometimes, we call something like ln -s to put them in the right place form post-get-deps hooks. Sometimes, they are referred to from pre-compile hook's Makefiles by doing something like -I $REBAR_DEPS_DIR/<protocol-defintions>.

[alavrik]

Allright. Let's take another shot at it.

The latest version is rebased on master and includes the following improvements:

• an example and description in rebar.config.sample
• the set of rebar subcommands recursively executed on raw dependencies is restricted to get-deps, update-deps, check-deps, list-deps and delete-deps
• make test passes without errors

[hyperthunk]

In hindsight, this would be very useful in other build environments, such as the RabbitMQ build system which currently can't be converted to rebar because of the non-OTP nature of so many of its components.

[hyperthunk]

In fact I'm now looking quite seriously at rebarizing the rabbit build, we cannot do it without this - we require things like our code generation library to be made available and it is python based.

[dizzyd]

So the only difference between check-deps and compile is that check_deps invokes save_dep_dirs/2? Why?

[alavrik]

@dizzyd Regrading compile, I just followed the original behavior where it was calling check-deps. check-deps was changed to call save_deps_dirs/2, because, unlike compile, it should work recursively on raw deps as well. The full list of deps will be returned later by postprocess/2 and raw deps will be processed recursively from there. The same applies to get-deps. I relied on the existing mechanism in rebar_core.erl to avoid (post-)processing non-raw deps twice.