bootstrap escript: exception error: no match of right hand side value error

[licaon-kter]

HEAD (5e22387)

$ ./bootstrap
...
===> Building escript for rebar...
escript: exception error: no match of right hand side value error
  in function  rebar_prv_escriptize:find_deps_of_deps/3 (/home/pi/rebar3/src/rebar_prv_escriptize.erl, line 256)
  in call from rebar_prv_escriptize:find_deps/2 (/home/pi/rebar3/src/rebar_prv_escriptize.erl, line 250)
  in call from rebar_prv_escriptize:escriptize/2 (/home/pi/rebar3/src/rebar_prv_escriptize.erl, line 110)
  in call from rebar_prv_escriptize:do/1 (/home/pi/rebar3/src/rebar_prv_escriptize.erl, line 89)
  in call from rebar_core:do/2 (/home/pi/rebar3/src/rebar_core.erl, line 155)
  in call from rebar_prv_do:do_task/5 (/home/pi/rebar3/src/rebar_prv_do.erl, line 75)
  in call from rebar_core:do/2 (/home/pi/rebar3/src/rebar_core.erl, line 155)
  in call from rebar3:run_aux/2 (/home/pi/rebar3/src/rebar3.erl, line 181)

Environment

Debian Buster, armv6, RPi1B 256Mb, Erlang/OTP 21 Erts 10.2.4

[licaon-kter]

Trying with 3.14.1

...
===> Analyzing applications...
===> Failed to get dependencies for /home/pi/rebar3/src/rebar_compiler_yrl.erl
{timeout,{gen_server,call,
                     [rebar_compiler_epp,
                      {resolve,"rebar_compiler.erl",
                               ["/home/pi/rebar3/include",
                                "/home/pi/rebar3/src"]}]}}
escript: exception throw: rebar_abort
  in function  rebar_utils:abort/0 (src/rebar_utils.erl, line 705)
  in call from rebar_compiler:analyze_app/3 (src/rebar_compiler.erl, line 194)
  in call from rebar_compiler:'-analyze_all/2-lc$^0/1-0-'/3 (src/rebar_compiler.erl, line 59)
  in call from rebar_compiler:analyze_all/2 (src/rebar_compiler.erl, line 59)
  in call from rebar_prv_compile:'-build_rebar3_apps/3-lc$^0/1-0-'/3 (src/rebar_prv_compile.erl, line 315)
  in call from rebar_prv_compile:build_rebar3_apps/3 (src/rebar_prv_compile.erl, line 331)
  in call from rebar_prv_compile:run_compilers/4 (src/rebar_prv_compile.erl, line 174)
  in call from rebar_prv_compile:compile/4 (src/rebar_prv_compile.erl, line 129)

[ferd]

please rm -rf _build before running the build again. There might be something odd there. Otherwise I'm not 100% sure what would be the problem right then. The version in master ought to be good, the error you see in 3.14.1 has been fixed (the timeout was removed).

The find_deps_of_deps call failing in the first snippet is weirder and there isn't a lot to go on there. If you can try running the whole thing again with DEBUG=1 or DIAGNOSTIC=1 to in hopes of generating a bit more output for us to debug this.

[licaon-kter]

please rm -rf _build before running the build again.

Right, forgot to do that, will retry on HEAD.

[licaon-kter]

DEBUG: debug1.txt

/LE: forgot to rm _build before DIAGNOSTIC, will redo

[licaon-kter]

DIAGNOSTIC: diagnostic.txt

[ferd]

From the diagnostics log:

===> new deps of <<"rebar">> found to be [<<"dialyzer">>,<<"getopt">>,
                                                 <<"erlware_commons">>,
                                                 <<"providers">>,
                                                 <<"bbmustache">>,
                                                 <<"ssl_verify_fun">>,
                                                 <<"certifi">>,
                                                 <<"cth_readable">>,
                                                 <<"relx">>,<<"cf">>,
                                                 <<"eunit_formatters">>]
===> processing <<"dialyzer">>
escript: exception error: no match of right hand side value error

it seems like the Dialyzer application is not built in your base Erlang install, and that is what crashes the build at this point. Rebar3 tries to analyze it to know if it actually needs including, but since it can't find it anywhere it can't figure out if it's part of the basic distribution and therefore errors out because it can't add it to the escript archive.

The solution would be to have a basic erlang install that comes with Dialyzer for rebar3 to build with dialyzer support.

If you want to work around that, downloading one of the pre-built images would likely work fine as long as you don't call rebar3 dialyzer.

[licaon-kter]

The solution would be to have a basic erlang install that comes with Dialyzer for rebar3 to build with dialyzer support.

I'd think the first solution would be for the bootstrap script to check, fail and output an error (or warning) if one of its dependencies is missing, at start.

Not sure what "a basic erlang install" is exactly, but erlang-dialyzer is not a dependency, at least on Debian: https://packages.debian.org/buster/erlang-nox

[ferd]

OS packages are always stripping away libraries that most Erlang users expect to be part of the standard distribution. We check for the commonly dropped ones (like ssl and crypto which comes from commonly missing dependencies like openssl or libressl), but assume an otherwise legit Erlang build.

We could arguably make a nicer check, but Linux packagers keep stripping different stuff at different time and we never really trust them. I still recall the times when Debian took a beta release of Erlang and stuck it for long term support and things haven't improved that greatly since then.

[licaon-kter]

Ok, that did it, once erlang-dialyzer was installed it built fine.

[licaon-kter]

Seeing this again on 26d929b with some random dep on armv6

Evaluating config script "/home/pi/rebar3/_build/default/lib/rebar/src/rebar.app.src.script"                ===> Expanded command sequence to be run: [app_discovery,install_deps,lock,compile]                                                         ===> Running provider: app_discovery                                  ===> Evaluating config script "/home/pi/rebar3/src/rebar.app.src.script"                                                                    ===> Found top-level apps: [rebar]                                            using config: [{src_dirs,["src"]},{lib_dirs,["apps/*","lib/*","."]}]                                                                ===> Running provider: install_deps                                   ===> Verifying dependencies...                                        ===> Getting definition for package bbmustache from repo hexpm (#{api_url => <<"https://hex.pm/api">>,name => <<"hexpm">>,                           repo_name => <<"hexpm">>,repo_organization => undefined,              repo_url => <<"https://repo.hex.pm">>,repo_verify => true,            repo_verify_origin => true})                                 ===> Hex get_package request failed: {error,                                                                       {failed_connect,
                                              [{to_address,
                                                {"repo.hex.pm",443}},
                                               {inet,
                                                [inet],
                                                {tls_alert,
                                                 {internal_error,
                                                  "TLS client: In state certify at ssl_handshake.erl:372 generated CLIENT ALERT: Fatal - Internal Error\n {unexpected_error,undef}"}}}]}}
===> Failed to update package bbmustache from repo hexpm
escript: exception error: no match of right hand side value
                 {error,
                     {rebar_app_utils,
                         {missing_package,<<"bbmustache">>,<<"1.12.2">>}}}
  in function  erl_eval:expr/5 (erl_eval.erl, line 453)
  in call from escript:eval_exprs/5 (escript.erl, line 872)
  in call from erl_eval:local_func/6 (erl_eval.erl, line 567)
  in call from escript:interpret/4 (escript.erl, line 788)
  in call from escript:start/1 (escript.erl, line 277)
  in call from init:start_em/1
  in call from init:do_boot/3

[ferd]

Yeah that commit is unrelated. The bootstrap script connecting there is talking to hex.pm and about certificate checks having issues.

This may depend on the OTP version you're using if you want further assistance, because behaviour changes across releases.

[licaon-kter]

This is Raspbian Bullseye so Erlang 23.2.6

[ferd]

Hm, I know we had issues with release candidates of 23. I just personally tested (on an intel linux) with 23.1 after having deleted my whole hex cache and local deps and it built fine as well. Looking at the tag for the version and the specific error:

{internal_error,
   "TLS client: In state certify at ssl_handshake.erl:372 generated CLIENT ALERT: Fatal - Internal Error\n {unexpected_error,undef}"}}}]}}

at least lets me find:

https://github.com/erlang/otp/blob/e8f9f260722457c48a801ba91f087cd986376fcd/lib/ssl/src/ssl_handshake.erl#L355-L372

This code is strictly related to validating the certificate chain, and the undef error tends to point towards a missing module or library. Specifically, the ssl library is built (because we're in it), and there are two direct external calls: public_key:pkix_path_validation and ssl_certificate:trusted_cert_and_paths (which also heavily relies on public_key).

If you open an erlang shell, what do you get trying to run the following command?

1> application:ensure_all_started(public_key).
{ok,[asn1,crypto,public_key]}

My guess is that the build you have is possibly missing cryptographic dependencies. Those are either due to libraries missing wholesale (e.g. public_key or crypto not being around) -- although this tends to be rare if the ssl library is also present -- or bindings to calls in C that would be required for Hex.pm's certificate chain that the Erlang ssl library expects to support but aren't wired in because the base release you're running has been built with too old of an openssl (or libressl) binding, maybe.

[licaon-kter]

As past comments here, I might have some missing deps, but this built fine on HEAD available on Nov 04.

Output:

 pi@raspberrypi:~$ erl shell
Erlang/OTP 23 [erts-11.1.8] [source] [smp:1:1] [ds:1:1:10] [async-threads:1]

Eshell V11.1.8  (abort with ^G)
1> application:ensure_all_started(public_key).
{ok,[asn1,crypto,public_key]}

[ferd]

The following pull requests were merged since November 1:

• git_resource: ignore stderr in get_patch_count when calling git rev-list (unused for bootstrapping, only hex deps are called)
• Fix NIF compilation on Apple M1. (template file, not used)
• Fix issue for duplicates options of 'parserfile' in 'yrl_opts' (no yrl files built in bootstrap)
• Support for all argument in unlock (not called in bootstrap)
• Support for all argument in upgrade (not called in bootstrap)
• Support for all argument in upgrade plugin (not called in bootstrap)
• add --relnames option to allow selectively building of multiple releases (not called in bootstrap)
• Bump bbmustache to support unicode attributes (not used in bootstrap)
• Proper respect of GLOBAL_CACHE_DIR

None of them should strictly have an impact on the bootstrap process; GLOBAL_CACHE_DIR would be the only one strictly making a difference during the bootstrap process, but only if you use global plugins and the value is set.

This is why I'm thinking this is something external, especially with the error taking place in code that's shipped with OTP, though we do inject options for proxies.

[licaon-kter]

I'll try to go back and bisect if needed.

[licaon-kter]

git bisect says...drum roll...

217d04388c368df49ba0864d48fec0f173b74682 is the first bad commit
commit 217d04388c368df49ba0864d48fec0f173b74682
Author: Fred Hebert <mononcqc@ferd.ca>
Date:   Thu Nov 11 14:47:42 2021 +0000

    Bump bbmustache to support unicode attributes

 rebar.config | 2 +-
 rebar.lock   | 6 +++---
 2 files changed, 4 insertions(+), 4 deletions(-)

[ferd]

that might just be because it finally needs to fetch a new version of the package (which actually exists) and highlights the underlying SSL issue. Otherwise, older builds keep fetching from the local disk cache.

[licaon-kter]

Right, which local folder should I delete?

[ferd]

that would be ~/.cache/rebar3/hex/ by default.

[licaon-kter]

Yup, cache nuked, no more compiles. What now?

[ferd]

Now the question is to figure out which part of your installation is messed up such that it won't validate TLS certs, fix that, and find a way that we could warn for it as well. I can't reproduce here, my install seems to work entirely fine and I'm betting this is environment specific.

[licaon-kter]

I did update to Bullseye at one point (before this moustache bump), so maybe erlang-* deps might be different now.

[licaon-kter]

Same issue on RPi3 arm64 Debian Bullseye

===> Running provider: install_deps                                   ===> Verifying dependencies...                                        ===> Getting definition for package bbmustache from repo hexpm (#{api_url => <<"https://hex.pm/api">>,name => <<"hexpm">>,                           repo_name => <<"hexpm">>,repo_organization => undefined,              repo_url => <<"https://cdn.jsdelivr.net/hex">>,repo_verify => true,
         repo_verify_origin => true})
===> Hex get_package request failed: {error,                                                                       {failed_connect,
                                              [{to_address,
                                                {"cdn.jsdelivr.net",443}},                                                                                                                 {inet,                                                                 [inet],                                                               {tls_alert,
                                                 {internal_error,                                                       "TLS client: In state wait_cert_cr at tls_handshake_1_3.erl:1428 generated CLIENT ALERT: Fatal - Internal Error\n {unexpected_error,undef}"}}}]}}               ===> Failed to update package bbmustache from repo hexpm              escript: exception error: no match of right hand side value                            {error,                                                                   {rebar_app_utils,
                         {missing_package,<<"bbmustache">>,<<"1.12.2">>}}}                                                                    in function  erl_eval:expr/5 (erl_eval.erl, line 453)
  in call from escript:eval_exprs/5 (escript.erl, line 872)             in call from erl_eval:local_func/6 (erl_eval.erl, line 567)           in call from escript:interpret/4 (escript.erl, line 788)
  in call from escript:start/1 (escript.erl, line 277)
  in call from init:start_em/1                                          in call from init:do_boot/3

...note the changed HEX_CDN just in case :(

[licaon-kter]

RPi3+ builds fine today, odd.

[ferd]

I'm wondering if there could be something specific about the certificate chain that was provided and the ciphersuites locally available over time. Hard to say for sure, but that sort of external change could explain the behaviour.

[licaon-kter]

Now errors out again on RPi3 at 81645a4

$ ./bootstrap
Dependency providers already exists
Dependency getopt already exists
Dependency cf already exists
Dependency erlware_commons already exists
Dependency certifi already exists
Evaluating config script "/home/pi/rebar3/_build/default/lib/rebar/src/rebar.app.src.script"
===> Verifying dependencies...
===> Failed to update package certifi from repo hexpm
escript: exception error: no match of right hand side value 
                 {error,
                     {rebar_app_utils,
                         {missing_package,<<"certifi">>,<<"2.9.0">>}}}
  in function  erl_eval:expr/5 (erl_eval.erl, line 453)
  in call from escript:eval_exprs/5 (escript.erl, line 872)
  in call from erl_eval:local_func/6 (erl_eval.erl, line 567)
  in call from escript:interpret/4 (escript.erl, line 788)
  in call from escript:start/1 (escript.erl, line 277)
  in call from init:start_em/1 
  in call from init:do_boot/3

As expected at fault is 01fa0c9 :)

/LE: removing cache/rebar3 we're back at square one.... {missing_package,<<"bbmustache">>,<<"1.12.2">>}}} :(

/LE2: DEBUG message

$ DEBUG=1 ./bootstrap
Dependency providers already exists
Dependency getopt already exists
Dependency cf already exists
Dependency erlware_commons already exists
Dependency certifi already exists
Evaluating config script "/home/pi/rebar3/_build/default/lib/rebar/src/rebar.app.src.script"
===> Expanded command sequence to be run: [app_discovery,install_deps,lock,compile]
===> Running provider: app_discovery
===> Evaluating config script "/home/pi/rebar3/src/rebar.app.src.script"
===> Found top-level apps: [rebar]
        using config: [{src_dirs,["src"]},{lib_dirs,["apps/*","lib/*","."]}]
===> Running provider: install_deps
===> Verifying dependencies...
===> Getting definition for package bbmustache from repo hexpm (#{api_url => <<"https://hex.pm/api">>,name => <<"hexpm">>,
         repo_name => <<"hexpm">>,repo_organization => undefined,
         repo_url => <<"https://repo.hex.pm">>,repo_verify => true,
         repo_verify_origin => true})
===> Hex get_package request failed: {error,
                                             {failed_connect,
                                              [{to_address,
                                                {"repo.hex.pm",443}},
                                               {inet,
                                                [inet],
                                                {tls_alert,
                                                 {internal_error,
                                                  "TLS client: In state certify at ssl_handshake.erl:372 generated CLIENT ALERT: Fatal - Internal Error\n {unexpected_error,undef}"}}}]}}
===> Failed to update package bbmustache from repo hexpm
escript: exception error: no match of right hand side value 
                 {error,
                     {rebar_app_utils,
                         {missing_package,<<"bbmustache">>,<<"1.12.2">>}}}
  in function  erl_eval:expr/5 (erl_eval.erl, line 453)
  in call from escript:eval_exprs/5 (escript.erl, line 872)
  in call from erl_eval:local_func/6 (erl_eval.erl, line 567)
  in call from escript:interpret/4 (escript.erl, line 788)
  in call from escript:start/1 (escript.erl, line 277)
  in call from init:start_em/1 
  in call from init:do_boot/3

[licaon-kter]

@tsloughter #2691 (comment) sounds like the same error? What was the fix? Hope not build from source :(

[ferd]

Yeah I mean we aren't changing anything. At this point there's work pending to just vendor dependencies, I figure this will solve your problems, but this isn't easily reproducible in other contexts.

[licaon-kter]

The odd thing is, you could say "your distro has a broken package" but then later it (last build was Jan 04) worked (no updates in the erlang packages afaik) then today doesn't work again. :(

[ferd]

Yeah. I mean part of me wonders if this is somewhat related to certificate chains or just mismatches between erlang versions as used and what's in the cache, but that alone is also really hard to figure out and wouldn't necessarily be so hard to reproduce.

[licaon-kter]

Today was a good day:

• cleanup, build, OK
• try again without clean up, FAIL
• clean up, build, OK
• try again without clean up, OK
• try again without clean up, OK
• try again without clean up, OK

I'd conclude it's not my system at fault :)

[ferd]

yeah that makes me wonder if there's something weird with the things returned over the network. I can't imagine that we'd generate random .beam files either.