rebar3 release crashes when bootstrapped with relx as a checkout dependency

[mxxk]

When rebar3 is bootstrapped with relx as a checkout dependency (symlink in _checkouts/relx), running rebar3 release crashes.

Background

When working on a relx fix and testing it through rebar3, I made relx a checkout dependency. However, it appears to have triggered a crash when using the bootstrapped rebar3.

I'm guessing that this is unique to rebar3 and its bootstrapping process, rather than a general issue with checkout dependencies.

Though I'm unfamiliar with the rebar3 internals, I'd be open to help out with this issue. 🙂

Environment

The environment below is from the Docker container repro in https://gist.github.com/mxxk/d986422a77806cb81bc760f4c97afa80. (The repro was originally obtained on macOS 10.15.7.)

$ /app/rebar3/rebar3 report release
Rebar3 report
 version 0.0.0
 generated at 2021-07-27T20:17:51+00:00
=================
Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: release
Entered as:
  release
-----------------
Operating System: x86_64-pc-linux-musl
ERTS: Erlang/OTP 24 [erts-12.0.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:no-native-stack]
Root Directory: /usr/local/lib/erlang
Library directory: /usr/local/lib/erlang/lib
-----------------
Loaded Applications:
bbmustache: 1.10.0
certifi: 2.6.1
cf: 0.3.1
common_test: 1.20.5
compiler: 8.0.2
crypto: 5.0.2
cth_readable: 1.5.1
dialyzer: 4.4.1
edoc: 1.0
erlware_commons: 1.5.0
eunit: 2.6.1
eunit_formatters: 0.5.0
getopt: 1.0.1
inets: 7.4.1
kernel: 8.0.1
providers: 1.8.1
public_key: 1.11.1
relx: 0.0.0
sasl: 4.1
snmp: 5.9.1
ssl_verify_fun: 1.1.6
stdlib: 3.15.2
syntax_tools: 2.6
tools: 3.5

-----------------
Escript path: /app/rebar3/rebar3
Providers:
  app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit get-deps help install install_deps list lock new path pkgs release relup report repos shell state tar tree unlock update upgrade upgrade upgrade version xref

Current behaviour

A repro of this issue is available in the form of a Dockerfile which fails at the rebar3 release step: https://gist.github.com/mxxk/d986422a77806cb81bc760f4c97afa80

Expected behaviour

• For rebar3 release to print out Thank you for developing with a checkout version of relx and run to completion 😛
  • (just kidding on the printing part)
• If there's a better way to develop a dependency of rebar3 which is to be tested with rebar3, perhaps there can be a small blurb about it on the rebar3 README or documentation website?

Workaround

A workaround for this which I ended up using was modifying rebar3's own rebar.config

-{relx, "4.4.0"},
+{relx, {git, "file:///local/path/to/relx", {branch, "development-branch"}}}

and running rebar3 unlock relx before ./bootstrap to pick up the latest local changes from relx.

Further considerations

• If relx was already fetched (e.g. with rebar3 get-deps) prior to it being symlinked to _checkouts, running ./bootstrap generates a rebar3 build which seemingly uses the non-development version of relx. (Isn't that odd?)
  • To reproduce:
    1. cd rebar3
    2. Remove _build and _checkouts, if they exist
    3. Use existing version of rebar3 to fetch dependencies: rebar3 get-deps
    4. Symlink _checkouts/relx to a development version
    5. ./bootstrap
    6. Run rebar3 release (using the rebar3 just bootstrapped) on another project
    7. Observe the output is not from the development version of relx

[mxxk]

Turns out the rootcause here is that when relx is a checkout dependency, its output directory is _build/<profile>/checkouts, not _build/<profile>/lib, so the lines

rebar3/rebar.config

Lines 30 to 31 in 1105538

	{escript_incl_extra, [{"relx/priv/templates/*", "_build/default/lib/"},
	{"rebar/priv/templates/*", "_build/default/lib/"}]}.

rebar3/rebar.config

Lines 70 to 72 in 1105538

	{escript_incl_extra, [
	{"relx/priv/templates/*", "_build/prod/lib/"},
	{"rebar/priv/templates/*", "_build/prod/lib/"}

do not result in relx/priv/templates being intended.

See #2602 for a fix

[ferd]

ah yeah that entirely makes sense. The priv dir stuff is a bit of a hack anyway.