Use Rebar to get, update and compile dependencies (and only dependencies) #503

amiramix · 2013-01-22T00:00:15Z

Zotonic is still compiled with the good old method using Emakefile

This Pull Request contains the minimal amount of changes to leverage Rebar dependency management without changing the compilation process of Zotonic itself.

Rationale:

When Zotonic is used as a part of a bigger application, the internal Zotonic dependencies (in Zotonic's deps folder) can clash with dependencies of the main application. Imagine the following folder structure:

main_app/deps/cowboy
main_app/deps/lager
main_app/deps/charreada
main_app/deps/zotonic/deps/cowboy
main_app/deps/zotonic/deps/lager
etc...

In this case cowboy (if used by Zotonic) and lager would be compiled and started twice from different paths.

Also, all Zotonic dependencies use Rebar and Rebar manages its own dependencies pretty well. For example Ranch is listed as dependency inside Cowboy, and both Cobwoy and Ranch are listed as dependencies inside Charrreada. When downloading dependencies with Rebar it will download Ranch and Cowboy only once to the main_app/deps folder (and NOT to local deps folders inside Cowboy or Charreada). It means everything is much simplified because all dependencies, no matter how complex the main app is, are only downloaded once into main_app/deps folder, and not to deps subfolders of the dependencies subfolders...

There is no reason to not leverage this simplicity when using Zotonic.

But on the other hand we don't want to compile Zotonic with Rebar because Emakefile is much quicker and is also used from z:m() (which wouldn't work when compiling Zotonic with Rebar).

Solution:

Add rebar.config and amend GNUmakefile so that all dependencies are managed by Rebar, but leave compiling Zotonic with the old method. The GNUmakefile has been simplified to contain a simple set of targets:

get-deps: Use Rebar to download all missing Zotonic dependencies; the old method with the use of git submodules works too - git submodules simply don't have to be used when Zotonic is compiled as a part of a bigger application

update-deps: Use Rebar to update dependencies

compile-deps: Use Rebar to compile dependencies, but only those which are present in Zotonic/deps sub-folder. If Rebar resolved Zotonic dependencies so that they are downloaded in the main_app/deps subfolder instead of main_app/deps/zotonic/deps then they won't be compiled by Zotonic's makefile but by the main_app's makefile !!

compile-zotonic: Old rules to compile Zotonic using Emakefile

all: compile-deps + compile-zotonic

docs, clean, clean_logs work as previously.

Please let me know if you have any questions or suggestions regarding this change.

…- Zotonic is still compiled with the good old method using Emakefile)

arjan · 2013-01-22T06:34:30Z

Nice change! I would even consider removing the git submodules entirely. IMHO they do more harm than good.

The only downside I see to using rebar is that it adds git as a build-time dependency. Any thoughts on how to prevent that?

kaos · 2013-01-22T08:05:22Z

Well, the git dependency is only for fetching the deps using the git:// protocol, right.

And, building from master (i.e. a git clone'd source tree), that should be fine.
For bundled source builds (i.e. downloaded tar balls) however, we could instead have deps pointing to http:// sources. That way it shouldn't need git to fetch them.

So, we would need some way to easily manage the rebar config, pointing to git deps from a dev repo, and to http deps on the release branch. Doable?

arjan · 2013-01-22T08:15:52Z

How do such HTTP deps look like in a rebar config?
because if you just change git:// to http:// it will still use git to clone the http:// one.

And we have the added issue that sub-dependencies might also depend on git being installed.

kaos · 2013-01-22T08:20:56Z

Ah, maybe that is a shortcoming of rebar then. I didn't look into it, just thinking it should work. (will look into it now...)

About deps having git deps.. maybe we need to host the http based deps, and make sure they are "release" friendly..
I think a tool is needed to ensure this. I envision a rebar extension, that can walk through the git deps, and fetch them into a output dir, along with modified rebar.config files that converts git to http in the process..
That way, it should be a simple command to update the dev rebar.config to a release one without any git in it.

arjan · 2013-01-22T08:23:40Z

I like that idea. zutils already has a z-archive command that does the above for git submodules. But I prefer using rebar, actually.

If we create a zip bundle like that, we can just list the dependencies in the rebar config without any source URL like this:

{deps, [lager, mochiweb, z_stdlib]}.

That will get them to build but does not assume anything about their source.

Arjan

kaos · 2013-01-22T08:56:39Z

Ah, even better. 👍

amiramix · 2013-01-22T11:54:36Z

I've created a simple escript command that reads rebar.config and extracts all Application names. That could be used to create a rebar.config file to use when doing a release.

However I have to say I don't quite see what is the problem. Unless someone specifically issues the command 'rebar get-deps' or 'rebar update-deps' rebar is not going to use the list of dependencies nor it is going to download anything using git or any other tool. Issuing just 'rebar compile' in a submodule doesn't cause rebar to download anything.

…e of the included file.

…bmit.

…ecause they come from the system. This prevents a lot of garbage being generated.

Modal title was rendered as text element. The problem was that htmlentities wasn't parsed at all: i.e. é -> é instead of é

This fixes a bug where an .mp3 file was renamed to .m2a on upload, because mimetypes returns multiple extensions for the mime type audio/mpeg. Instead of taking the first extension found, it checks whether the original filename has already an extension that is allowed.

amiramix · 2013-01-24T14:25:27Z

Please let me know if you require any changes to my pull-request needed before integrating it or if you have any questions or doubts.

arjan · 2013-01-24T14:30:13Z

Two things:

Remove the git submodules because that is the same as using rebar deps
Make "rebar compile" return an error message (or let it call "make"); if ppl see a rebar.config they expect to compile it with rebar as well.

After that this is good to merge imho :) 👍

amiramix · 2013-01-24T14:35:14Z

Any idea how I can intercept "rebar compile" to not let it compile Zotonic? I looked a little bit around but couldn't find anything about it. I don't think it's possible. It's another limitation of rebar I guess. I may ask on the rebar forum though, I heard there is one.

amiramix · 2013-01-24T14:40:37Z

On a second thought, rebar doesn't do anything special during compilation and as long as Zotonic follows OTP there shouldn't be any reason why ppl shouldn't want to compile Zotonic with Rebar. I know it currently it doesn't compile because some of the includes it can't find, but maybe the solution is to make Zotonic compilable by Rebar as an option if someone really wants to?

amiramix · 2013-01-24T15:04:16Z

Another problem is that when zotonic is itself a dependency of another application, rebar refuses to compile Zotonic (by doing e.g. cd deps/zotonic && rebar compile) because it can't find the dependencies. It can't, of course, because they have been already downloaded as dependencies of the main project, not Zotonic dependencies. There is no such concept like compile just one dependency in Rebar.

I have to say that Rebar sucks. It may be fine for small projects but to use it with a bigger projects with multiple dependencies it is pain in the ... I am glad Zotonic doesn't depend on Rebar.

arjan · 2013-01-24T15:07:15Z

Heheh. But do you still agree with the approach for this pull request then?

amiramix · 2013-01-24T15:13:51Z

Well, yes, because since all dependencies use Rebar I still think Rebar should be used to manage them. I just don't like Rebar's approach - if you use me to compile one project then it's best for you if you use me for all the other your projects too. It's like cancer. Everything should be rebarized or nothing.

In this case the problem I guess is with Dependencies of Zotonic dependencies. If I have a bigger application with many dependencies, Zotonic being one of them, and those apps may have their own dependencies, then when I want to create a deployment I'd like to have all those dependencies in a single folder, so they are not duplicated (as I stated originally). If all those dependencies use Rebar why don't use it to manage them then?

mmzeeman · 2013-01-24T15:30:50Z

Recently found that out too… I wanted to continue development so I worked around rebar. Very annoying.

kaos · 2013-01-24T15:34:27Z

GNUmakefile

+
+# Phony rule to compile a particular dependency with Rebar
+compile-rule/%: $(REBAR)
+	cd $* && $(REBAR) compile


I see an issue here, if REBAR points to ./rebar and the dep doesn't have rebar itself, it will break. Better is to define REBAR with an absolute path, that way the dep doesn't have to have rebar itself.

You are right, thanks, will be posting a fix shortly.

mmzeeman · 2013-01-24T15:44:53Z

I solved it by calling rebar for each sub-project with a fixed deps dir and handle all the deps from a rebar config in the main project. What happens then is that rebar compiles multi deps, like z_stdlib, multiple times.

What I also found odd, from a security standpoint, is that each of those project ship with binary rebar executables.

kaos · 2013-01-24T15:47:43Z

OK, so rebar doesn't scale well. I think that zotonic's current setup doesn't scale well either (as in having zotonic as a dependency in a larger setup).
With that in mind, trying to solve the latter might solve the former as well (not solve per se, but get rid of the need).

So, just brainstorming here, one way to make zotonic fit in a bigger picture would be to defer the dependency parts for zotonic. That would leave us with a split of the current zotonic into a zotonic-lib, and a zotonic-app. The zotonic-lib has all the functionality, and zotonic-app simply ties the whole thing together with the required deps and all. So if zotonic is to be used as a dep in another project, they can point to zotonic-lib, and take care of the needed deps them selves.

Especially if all deps are OTP apps, it should be a breeze.

Just to clarify my thoughts (in case my writing was a bit fuzzy):

ERL_LIBS/
   lager-1.0.0/
   bert-master/
   zstdlib-master/
   zotonic_app-0.9.1/    <-- simple app just to kick start the zotonic_lib
   zotonic_lib-0.9.1/      <-- holds all code, modules etc that is zotonic today 
                     except the deps; it only lists the required deps in the .app file
   otherstuff-x.y.z/

Edit:
Ah, missed to say, that it would be up to the zotonic_app to make sure we have and build the required deps.

…ject

amiramix · 2013-01-24T17:14:13Z

@kaos
I think it depends how you define a dependency for Zotonic. Zotonic itself follows the OTP principle in having the src, ebin, priv and include libraries. I think what's spoiling the picture are all the other folders like models, modules, controllers, etc. I am not sure if they break OTP. Maybe it would be enough to just move them under the src directory?

Also, I am not sure what you mean by zotonic-lib. Zotonic must be itself as an app. In OTP when another project depends on Zotonic it just lists it as one of the dependencies, e.g. {applications, [kernel, stdlib, cowboy, ibrowse]}. Then OTP starts those apps automatically, in case of Zotonic by calling zotonic:start().

Zotonic can be listed in the release file too:

{release,
[ {kernel, (...)
,{compiler, "4.8.2"}
,{syntax_tools, "1.6.9"}
,{lager, "2.0.0"}
,{webzmachine, "1.8.1"}
,{zotonic, "0.10-dev"}
]}.

I don't quite see how you would like to separate zotonic modules internally into app and lib parts.

amiramix · 2013-01-24T17:15:43Z

@mmzeeman
I understand your approach, but for that you would need to change the zotonic's rebar.config (to specify a custom deps folder)?

mmzeeman · 2013-01-24T19:49:48Z

It was just what I did to get going. I wanted to program and not endlessly fiddle with rebar.this and such.so config.

It was obvious it doesn't do dependency management very well (cough), so i used this flat dep structure.

If you see Zotonic as a dep, then all its deps should be listed in your own deps dir and you should get them yourself.

Either by using git sub modules, rebar deps or just using curl in a makefile somewhere.

You can pass the location of this big flat deps dir to the rebar invocation of the sub modules during the build. Probably not the way it should work. This was just a quick strategy to get going.

All in all very unsatisfactory IMHO.

Maas

On 24 jan. 2013, at 18:15, amiramix notifications@github.com wrote:

@mmzeeman
I understand your approach, but for that you would need to change the zotonic's rebar.config (to specify a custom deps folder)?

—
Reply to this email directly or view it on GitHub.

kaos · 2013-01-24T21:11:22Z

@amiramix
I felt my post would be confusing. I'll try to elaborate my idea a bit, see if it helps :p

First, the zotonic-lib name is a bit misleading, as it is meant to be an OTP app as well.
The modules dir need to move, I think, to be fully OTP-compliant, but that is beside the point.

My discussion was focusing on getting rid of the deps dir. The dependencies can still be listed in the .app file,
but when building zotonic-lib, all dependencies need to be met (and built) already. This is the job for zotonic-app,
which itself is another OTP app. It has a single dependency (in the .app file), namely the zotonic-lib. However, when building zotonic-app, it makes sure that the deps needed by zotonic-lib are also present, and built.

So, at build time, zotonic-app manages the deps, and at runtime zotonic-lib ensures they are all started (by listing them in it's .app file).

This way, in case zotonic is to be used as "just another dep" in a larger app that already has all (or some of) the deps needed by zotonic, already with its own build system, it can just drop in zotonic-lib as another dep and leave zotonic-app out of the way. In case not all zotonic deps were already met, it needs to add those, of course, and can peek at the deps pulled in by zotonic-app to find out which deps are needed (and which versions etc).

It's getting late here, and maybe it's not such an great idea, so I'll stop typing now.. ;)

amiramix · 2013-01-24T22:24:31Z

I see, but this is exactly why I did this change that is in this pull request. Basically, when Rebar is used to manage Zotonic dependencies it downloads all dependency dependencies into one place. I tried to explain that in my original comment. There is no need to split Zotonic up into separate lib and app applications.

So, by example, this is rebar.config of the main project:

{deps,
 [{charreada, ".*", {git, "git://github.com/yoonka/charreada.git", "master"}},
  {zotonic, ".*", {git, "git://github.com/zotonic/zotonic.git", "master"}}]}.

Now, when I do rebar list-deps I get:

==> Entering directory `/usr/home/g/main/project/deps/charreada'
==> Entering directory `/usr/home/g/main/project/deps/cowboy'
==> Entering directory `/usr/home/g/main/project/deps/ranch'
==> ranch (list-deps)
==> Leaving directory `/usr/home/g/main/project/deps/ranch'
==> cowboy (list-deps)
ranch REV 0.6.0 git://github.com/extend/ranch.git
==> Leaving directory `/usr/home/g/main/project/deps/cowboy'
==> Entering directory `/usr/home/g/main/project/deps/ibrowse'
==> ibrowse (list-deps)
==> Leaving directory `/usr/home/g/main/project/deps/ibrowse'
==> charreada (list-deps)
cowboy REV master git://github.com/extend/cowboy.git
ibrowse REV master git://github.com/cmullaparthi/ibrowse.git
==> Leaving directory `/usr/home/g/main/project/deps/charreada'
==> Entering directory `/usr/home/g/main/project/deps/zotonic'
==> Entering directory `/usr/home/g/main/project/deps/bert'
==> bert (list-deps)
==> Leaving directory `/usr/home/g/main/project/deps/bert'
==> Entering directory `/usr/home/g/main/project/deps/dh_date'
==> dh_date (list-deps)
==> Leaving directory `/usr/home/g/main/project/deps/dh_date'
==> Entering directory `/usr/home/g/main/project/deps/gen_smtp'
==> gen_smtp (list-deps)
==> Leaving directory `/usr/home/g/main/project/deps/gen_smtp'
==> Entering directory `/usr/home/g/main/project/deps/lager'
==> lager (list-deps)
==> Leaving directory `/usr/home/g/main/project/deps/lager'
==> Entering directory `/usr/home/g/main/project/deps/mimetypes'
==> mimetypes (list-deps)
==> Leaving directory `/usr/home/g/main/project/deps/mimetypes'
==> Entering directory `/usr/home/g/main/project/deps/ua_classifier'
==> ua_classifier (list-deps)
==> Leaving directory `/usr/home/g/main/project/deps/ua_classifier'
==> Entering directory `/usr/home/g/main/project/deps/webzmachine'
==> webzmachine (list-deps)
==> Leaving directory `/usr/home/g/main/project/deps/webzmachine'
==> Entering directory `/usr/home/g/main/project/deps/z_stdlib'
==> z_stdlib (list-deps)
==> Leaving directory `/usr/home/g/main/project/deps/z_stdlib'
==> zotonic (list-deps)
bert REV master git://github.com/zotonic/bert.erl.git
dh_date REV master git://github.com/zotonic/dh_date.git
gen_smtp REV master git://github.com/zotonic/gen_smtp.git
lager TAG 1.0.0 git://github.com/basho/lager.git
mimetypes REV master git://github.com/zotonic/mimetypes.git
ua_classifier REV master git://github.com/zotonic/ua_classifier.git
webzmachine REV master git://github.com/zotonic/webzmachine.git
z_stdlib REV master git://github.com/zotonic/z_stdlib.git
==> Leaving directory `/usr/home/g/main/project/deps/zotonic'
==> mainproject (list-deps)
charreada REV master git://github.com/yoonka/charreada.git
zotonic REV master git://github.com/zotonic/zotonic.git

As you see, it lists all dependencies of all the dependencies, including Zotonic. When I do rebar get-deps it downloads all of them into mainproject/deps leaving Zotonic/deps empty (apart from iconv and mochiweb). I wrote the new Zotonic makefile in such a way that it compiles only what is present in Zotonic/deps so if Rebar downloaded Zotonic dependencies into the main deps folder they won't be compiled (they will be compiled by the makefile of the main project).

Hope that makes sense?

kaos · 2013-01-25T09:14:15Z

Ah, nice!

I didn't get that part from your initial comment :)

Definitely a step in the right direction then 👍

amiramix · 2013-01-25T11:00:53Z

I assumed that everyone who is using rebar already knows about this feature and I am the last one to learn that it works like that, as I myself discovered it only a couple of weeks ago :)

Anything else is needed in my pull request before considering integrating it into Zotonic project?

…ot in the default folder

kaos · 2013-03-05T12:16:21Z

Replaced by #515.

Use Rebar to get, update and compile dependencies (only dependencies …

6f3a8c3

…- Zotonic is still compiled with the good old method using Emakefile)

mworrell and others added 9 commits January 22, 2013 16:37

core: allow an included file to use extends/overrules within the scop…

bbcc9d8

…e of the included file.

mod_survey: fix for yes/no and true/false questions with immediate su…

97ab471

…bmit.

Module indexing optimizations.

4e3e2f0

Prevent the usage of filename:join, we can assume filenames are ok, b…

6aae557

…ecause they come from the system. This prevents a lot of garbage being generated.

doc: Add build: tag to list of prefixes.

02560b7

doc: Add translations: tag to list of prefixes.

3a865bd

doc: rename translations: to translation:.

2b586c4

mod_base: Modal html title in z.dialog

bbf2881

Modal title was rendered as text element. The problem was that htmlentities wasn't parsed at all: i.e. é -> é instead of é

kaos reviewed Jan 24, 2013
View reviewed changes

amiramix added 2 commits January 24, 2013 16:54

Delete git submodules and allow rebar to compile the main Zotonic pro…

8a85abc

…ject

Allow building dependencies with locally downloaded Rebar

94cdd2b

amiramix added 3 commits January 25, 2013 12:55

Fix relative lib includes so that Zotonic can be compiled with deps n…

a5fcb70

…ot in the default folder

Update controller_static_pages documentation

1cf5929

Merge branch 'manage-dependencies-with-rebar' into static-pages-update

60a2ab9

amiramix mentioned this pull request Feb 2, 2013

Use Rebar to manage and compile dependencies and optionally compile Zotonic #515

Merged

kaos closed this Mar 5, 2013

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Rebar to get, update and compile dependencies (and only dependencies) #503

Use Rebar to get, update and compile dependencies (and only dependencies) #503

amiramix commented Jan 22, 2013

arjan commented Jan 22, 2013

kaos commented Jan 22, 2013

arjan commented Jan 22, 2013

kaos commented Jan 22, 2013

arjan commented Jan 22, 2013

kaos commented Jan 22, 2013

amiramix commented Jan 22, 2013

amiramix commented Jan 24, 2013

arjan commented Jan 24, 2013

amiramix commented Jan 24, 2013

amiramix commented Jan 24, 2013

amiramix commented Jan 24, 2013

arjan commented Jan 24, 2013

amiramix commented Jan 24, 2013

mmzeeman commented Jan 24, 2013

kaos Jan 24, 2013

amiramix Jan 24, 2013

mmzeeman commented Jan 24, 2013

kaos commented Jan 24, 2013

amiramix commented Jan 24, 2013

amiramix commented Jan 24, 2013

mmzeeman commented Jan 24, 2013

kaos commented Jan 24, 2013

amiramix commented Jan 24, 2013

kaos commented Jan 25, 2013

amiramix commented Jan 25, 2013

kaos commented Mar 5, 2013

Use Rebar to get, update and compile dependencies (and only dependencies) #503

Use Rebar to get, update and compile dependencies (and only dependencies) #503

Conversation

amiramix commented Jan 22, 2013

arjan commented Jan 22, 2013

kaos commented Jan 22, 2013

arjan commented Jan 22, 2013

kaos commented Jan 22, 2013

arjan commented Jan 22, 2013

kaos commented Jan 22, 2013

amiramix commented Jan 22, 2013

amiramix commented Jan 24, 2013

arjan commented Jan 24, 2013

amiramix commented Jan 24, 2013

amiramix commented Jan 24, 2013

amiramix commented Jan 24, 2013

arjan commented Jan 24, 2013

amiramix commented Jan 24, 2013

mmzeeman commented Jan 24, 2013

kaos Jan 24, 2013

Choose a reason for hiding this comment

amiramix Jan 24, 2013

Choose a reason for hiding this comment

mmzeeman commented Jan 24, 2013

kaos commented Jan 24, 2013

amiramix commented Jan 24, 2013

amiramix commented Jan 24, 2013

mmzeeman commented Jan 24, 2013

kaos commented Jan 24, 2013

amiramix commented Jan 24, 2013

kaos commented Jan 25, 2013

amiramix commented Jan 25, 2013

kaos commented Mar 5, 2013