Fresh-clone builds fail with "No rule to make target gen/nqp-version', needed by blib/Perl6/ModuleLoader.moarvm'."

[zoffixznet]

After changes done in the past 24 hours—possibly the build chain work performed by @niner —the clean rebuild (or fresh clone) of rakudo fails with:

Using /home/zoffix/R/install/bin/nqp-m (version 2018.03-35-gce0263e3f / MoarVM 2018.03-29-g621ca3c22).
Cleaning up ...

You can now use 'make' to build Rakudo.
After that, 'make test' will run some tests and
'make install' will install Rakudo.
/home/zoffix/perl5/perlbrew/perls/perl-5.26.1/bin/perl tools/build/check-nqp-version.pl /home/zoffix/R/install/bin/nqp-m
make: *** No rule to make target `gen/nqp-version', needed by `blib/Perl6/ModuleLoader.moarvm'.  Stop.
make: *** Waiting for unfinished jobs....
The spawned command 'make' exited unsuccessfully (exit code: 2)
  in method build-rakudo at /home/zoffix/zscript/bin/../lib/RDev.pm6 (RDev) line 239
  in sub MAIN at /home/zoffix/zscript/bin/z line 22
  in block <unit> at /home/zoffix/zscript/bin/z line 148

The above is the output from Z-Script's z z command that pulls and does clean rebuild of MoarVM/nqp/rakudo. The same problem occurs with z init ~/Dir that does a clean clone and build of the three products.

I could NOT reproduce the issue with just a rakudo clone+ build:

cd $(mktemp -d) && 
git clone https://github.com/rakudo/rakudo/ . && 
perl Configure.pl --gen-moar --gen-nqp --backends=moar && 
make && make test && make install

However, the problem does appear (without Z-Script involvement) if the three products are cloned
and built separately with this set of commands:

cd $(mktemp -d)
git clone https://github.com/MoarVM/MoarVM
git clone https://github.com/perl6/nqp
git clone https://github.com/rakudo/rakudo
RAK_INSTALL_DIR=$(pwd)/install
mkdir "$RAK_INSTALL_DIR"

cd MoarVM
perl Configure.pl --prefix="$RAK_INSTALL_DIR"
make -j24
make install

cd ../nqp
perl Configure.pl --prefix="$RAK_INSTALL_DIR" --backends=moar
make -j24
make test
make install

cd ../rakudo
perl Configure.pl --prefix="$RAK_INSTALL_DIR" --backends=moar &&
make -j24 &&
make install || echo 'FAILED TO INSTALL'

---

It's worth noting that running make clean after the build failure and re-running rakudo's make/make install avoids the problem.

After a successful build, there is an un-tracked gen/nqp-version file created in rakudo's directory with some sha hash in it. As long as that file exists, rebuilding everything with z z command works, but if I delete that file, the issue comes back again.

Just a thought: is it possible whatever creates that file assumes the ./rakudo/nqp/MoarVM nested checkout of repos that you get if you let rakudo's build auto-clone nqp and MoarVM?

[niner]

Oooooooh..... it's the -j24 on the make. tools/build/check-nqp-version.pl does run and produces a correct nqp-version file, but at that point it's already too late, because while it was running, make tried to check the dependency for other parts. Just running make again will therefore succeed. I didn't notice before because I never run make for rakudo with multiple jobs as it doesn't make a difference anyway.

The following patch could fix that, but then it could miss the need for a rebuild as it's updating gen/nqp-version after checking that dependency for other targets. I think, there's some special make target for forcing non-parallel build of some targets. That's gonna be the real solution.

diff --git a/tools/build/Makefile-Moar.in b/tools/build/Makefile-Moar.in
index 23171dcbd..9de40bf5e 100644
--- a/tools/build/Makefile-Moar.in
+++ b/tools/build/Makefile-Moar.in
@@ -97,7 +97,9 @@ M_HARNESS6_WITH_FUDGE = $(M_HARNESS6) --fudge
 
 M_RUN_CLEAN_TARGET_FILES = $(M_RUN_PERL6) -e "for @*ARGS.head(*-1) { given (@*ARGS[*-1] ~ '/' ~ .IO.basename.Str) { say 'rm -f ' ~ .Str; .IO.unlink if .IO.e } }"
 
-m-all: check_nqp_version $(PERL6_MOAR) $(SETTING_MOAR) $(SETTING_D_MOAR) $(R_SETTING_MOAR) $(M_RUNNER) $(PERL6_DEBUG_MOAR) $(M_DEBUG_RUNNER) @m_all@
+m-all: gen/nqp-version $(PERL6_MOAR) $(SETTING_MOAR) $(SETTING_D_MOAR) $(R_SETTING_MOAR) $(M_RUNNER) $(PERL6_DEBUG_MOAR) $(M_DEBUG_RUNNER) @m_all@
+
+gen/nqp-version: check_nqp_version
 
 check_nqp_version: tools/build/check-nqp-version.pl
        $(PERL5) tools/build/check-nqp-version.pl $(M_NQP)

[zoffixznet]

Confirming that getting rid of -j on rakudo's make in ZScript avoids the problem.

[dogbert17]

Works without issues for me using:

make -j6

System info:

dogbert@dogbert-VirtualBox ~/repos/rakudo $ uname -a
Linux dogbert-VirtualBox 4.10.0-38-generic #42~16.04.1-Ubuntu SMP Tue Oct 10 16:32:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
dogbert@dogbert-VirtualBox ~/repos/rakudo $ make -v
GNU Make 4.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.