Working with versioned deps can be a nightmare

[oubiwann]

This is a pretty big issue, and one that really needs to have some discussion on the mail list. However, I want to record it here so we don't lose track of it.

The versioning of LFE-based projects in their respective rebar.config files was solely focused on providing a development environment where updates in one project -- even breaking ones -- would not disturb developers of other projects.

The solution to that problem resulted in something that's fairly rigid, and having worked with Docker to create a custom app based on @rvirding's latest work in the develop branch, I had to essentially gut the rebar.config files for each of the deps and manually include them in the sample project.

There has got to be a better way.

I'll bring this up on the mail list ...

[oubiwann]

There is a good conversation in issue #111 that relates to this.

[oubiwann]

In issue #111, @rvirding made a request about addressing this issue, in particular asking if we could have options in lfetool that would allow users to override the default version-pinning (both in the generated projects, as well as in the deps of those generated projects).

It would be fairly easy to implement this if the following rebar issue was fixed:

• https://github.com/rebar/rebar/issues/170

However, it may be a while before that happens. We may have to work around rebar, as I said in my reply to Robert:

There are a couple of things that come to mind ... but I think we'd have to:

• provide a way of indicating to lfetool that a project should not use the default, version-pinned rebar deps, and
• get the deps to also ignore their own version-pinned deps

I'm not sure if this can be done within the confines of rebar's current feature set. We may have to implement this with bash, LFE, or Erlang.

[oubiwann]

Hrm, working on a rebar work-around for over-riding the default could be the first steps towards addressing this feature request:

• Support user customisation of LFE via options lfe/lfe#103

lfetool could parse an lfe.config file that has a top-evel entry for deps like the rebar.config deps. If defined here, it could override rebar. For now, it would only support a minimal structure (nothing like the varieties of options that rebar allows in its deps. The simplest form of deps support in rebar (beside the empty list, of course) is just a list of project name atoms, e.g. [lfe, lutil, ltest, yaws].

Hrm, I guess we'd need to support [{lfe, "master"}, ...] so users could select which branch to use.

[oubiwann]

I'm working on a draft of this idea with the following:

• Ticket: Create functions that support a first draft of lfe.config lfex/lutil#23
• Branch: https://github.com/lfex/lutil/tree/lfe-config

[oubiwann]

So how this might work:

• Add support for the command lfetool download deps
• Update all the lfetool skeleton project templates to use lfetool download deps instead of rebar get-deps
• When that is run, if the ~/.lfe/deps/lutil directory doesn't exist, lfetool will clone lutil into it
• With ~/.lfe/deps/* added to ERL_LIBS, lfetool will then call (lutil-cfg:clone-deps)
• That will only have an effect if 1) #(project (#(deps list-of-deps))) exists in either ~/.lfe/lfe.config or ./lfe.config, and 2) if those listed deps haven't already been git cloned
• After lfetool runs (lutil-cfg:clone-deps), it will then run rebar get-deps

This will allow for the following scenarios:

• Users can define project deps in ~/.lfe/lfe.config that will override ALL rebar deps (e.g., "No matter what, I want rvirding/lfe master used for every single LFE project, unless I choose to override this in a local, project lfe.config.")
• Users can selectively override rebar deps on a per-project basis with the deps listed in ./lfe.config
• By not having deps in either ~/.lfe/lfe.config or ./lfe.config, users can let rebar do all the work, keeping all versions safely pinned. Safe like cold, iron bars. In a jail.

So in the end, nothing really has to be changed for the end users and no crazy options need to be added to lfetool. Things will continue working as they have been.

But should a user want to override versions of deps, they just need to create anlfe.config file and keep using the updated make targets or use the lfetool download deps command.

[oubiwann]

Note to self: lfetool calls to rebar should remember Windows:

which rebar.cmd >/dev/null 2>&1 && rebar.cmd get-deps || rebar get-deps

[oubiwann]

Okay, I've got a very early implementation of the lfetool download deps command in place.

The command is documented here:

• https://github.com/lfe/lfetool/blob/dev-v1/doc/manual/download.md

Testing Plan

I need to do a bunch of testing with it:

• test with no config files (remove ~/.lfe/lfe.config and ./lfe.config
  • filed lutil-cfg: edit git 'fatal' message lfex/lutil#26; fixed
• test with only ~/.lfe/lfe.config
  • empty (zero bytes)
    • filed lutil-cfg: empty configfile throws an error lfex/lutil#27; fixed
  • with non-tuples in top-level data
  • with config options, but not one for project deps
  • with project deps option, but no other options
  • with project but no deps option and no other options
    • filed lutil-cfg: Error with empty project config option lfex/lutil#28; fixed
  • with project but no deps option but with other options
    • filed lutil-cfg: Error with non-deps project config option lfex/lutil#29; fixed
  • with both project deps and other options
• test with only ./lfe.config
  • empty (zero bytes)
    • filed lutil-cfg: empty configfile throws an error lfex/lutil#27; fixed
  • with non-tuples in top-level data
  • with config options, but not one for project deps
  • with project deps option, but no other options
  • with project but no deps option and no other options
  • with project but no deps option but with other options
  • with both project deps and other options
• test with both ./lfe.config and ~/.lfe/lfe.config
  • with no deps in either
  • with deps in ./lfe.config but not in ~/.lfe/lfe.config
  • with deps in ~/.lfe/lfe.config but not in ./lfe.config
  • with different deps in both
  • with some of the same deps in both
    • filed lutil-cfg: local config overrides all of the global deps lfex/lutil#30; fixed
    • turns out this issue will require a bit of a refactor; I'm adding unit tests to make this more pain-free; may need to re-run through all of these QA tests, though ...
  • with all of the same deps in both
• test projects with an lfe.config file
  • yes, as feared, the impact on other rebar-using make targets is pretty big; these will each need tickets of their own
  • ensure that make compile* still works
    • filed Ensure that "make compile" works around rebar when lfe.config is present #146; fixed
  • ensure that make repl* still works
    • filed Ensure that "make repl" works around rebar when lfe.config is present #147; fixed
  • ensure that make check* still works
    • filed Ensure that "make check" works around rebar when lfe.config is present #148; fixed
  • make sure that the update make get-deps target (which calls lfetool download deps) works appropriately during the lfetool bootstrap phase of lutil in ~/.lfe/deps
  • bump the versions of lutil, ltest, lfest, and exemplar (all projects which are included in various lfetool skeleton projects)
  • release said projects
  • update lfetool dev-v1 rebar templates to use these new versions
  • build, commit, and push a new lfetool with all the latest changes so it can be used for testing
• Test on fresh systems, using Docker:
  • filed: lfetool call to "rebar get-deps" fails occasionally #149; fixed
  • on Ubuntu 14.40, test:
    • create a library project to test lutil, ltest (unit tests will be run automatically)
    • run unit tests and check versions with and without an lfe.config file
    • create a web app project to test lutil, ltest, lfest, and examplar (unit tests will be run automatically)
    • run unit tests and check versions with and without an lfe.config file
  • on Debian Jessie 8.0, test:
    • create a library project to test lutil, ltest (unit tests will be run automatically)
    • run unit tests and check versions with and without an lfe.config file
    • create a web app project to test lutil, ltest, lfest, and examplar (unit tests will be run automatically)
    • run unit tests and check versions with and without an lfe.config file
  • on CentOS 7, test:
    • create a library project to test lutil, ltest (unit tests will be run automatically)
    • run unit tests and check versions with and without an lfe.config file
    • create a web app project to test lutil, ltest, lfest, and examplar (unit tests will be run automatically)
    • run unit tests and check versions with and without an lfe.config file

Notes for Testing

lfe.config

./lfe.config file for testing:

#(project (#(deps (#("rvirding/lfe" "develop")))))

Docker

Debian and Ubuntu Container Setup

Host:

$ docker run -t -i debian /bin/bash

Guest:

$ apt-get update && apt-get install -y \
    libpam0g-dev erlang git curl build-essential

CentOS Container Setup

Host:

$ docker run -t -i centos /bin/bash

Guest:

$ rpm -Uvh \
http://download.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
$ yum -y groupinstall 'Development Tools' && \
  yum -y install wget erlang pam-devel

Setup for All Containers

rebar

$ curl -L -o /usr/local/bin/rebar \
    https://github.com/rebar/rebar/releases/download/2.5.1/rebar
$ chmod 755 /usr/local/bin/rebar

lfetool

The installation for lfetool on these Docker images:

$ curl -L -o ./lfetool \
    https://raw.github.com/lfe/lfetool/dev-v1/lfetool
$ bash ./lfetool install
$ lfetool -x

Testing for All Containers

$ lfetool new library lib-1
$ cd lib-1
$ make get-version; rm -rf deps/lfe
$ echo '#(project (#(deps (#("rvirding/lfe" "develop")))))' > lfe.config
$ make check; make get-version
$ cd ..
$ lfetool new yaws bootstrap web-1
$ cd web-1
$ make get-version; rm -rf deps/lfe
$ echo '#(project (#(deps (#("rvirding/lfe" "develop")))))' > lfe.config
$ make check; make get-version

Note that if you get a project creation failure (e.g., dependencies not found), try again:

$ rm -rf lib-1
$ lfetool new library lib-1

or

$ lfetool new library lib-2