Rakudo no longer builds properly on OpenBSD #2904

Kaiepi · 2019-05-17T10:51:23Z

The Problem

The new build scripts try to build a relocatable executable by default on OpenBSD when that doesn't work.

Expected Behavior

They should build a non-relocatable executable.

Actual Behavior

They don't. Passing --no-relocatable also doesn't work without manually editing the makefile since it gets passed to it as --no-relocatable="1" when it doesn't take a value, making one of the scripts called by make throw. Also, Configure.pl incorrectly states that you should use gmake instead of make.

Steps to Reproduce

Run ./Configure.pl && make install, then try to run perl6.

Environment

Operating system:
OpenBSD bastille.kennel.qt 6.5 GENERIC.MP#8 amd64
Compiler version (perl6 -v):
This is Rakudo version 2019.03.1-405-g7a7a23df6a built on MoarVM version 2019.03-111-g24b3ef1ad
implementing Perl 6.d.

The text was updated successfully, but these errors were encountered:

vrurg · 2019-05-17T13:47:28Z

With regard to gmake – do you have it in your path? If so, then this is the recommended make to use. But if you don't then this is a problem.

vrurg · 2019-05-17T14:07:31Z

I'm installing openBSD in a VM. Meanwhile may I ask for the output of the failing script?

Kaiepi · 2019-05-17T14:11:00Z

With regard to gmake – do you have it in your path? If so, then this is the recommended make to use. But if you don't then this is a problem.

I do. I don't think it should be the recommended make on *BSD though since it's not installed by default.

I'm installing openBSD in a VM. Meanwhile may I ask for the output of the failing script?

bastille% ./Configure.pl --backends=moar --prefix=$HOME/.perl6 --no-relocatable
Using /home/morfent/.perl6/bin/nqp-m (version 2019.03-83-g0adfd6ada / MoarVM 2019.03-111-g24b3ef1ad).
Cleaning up ...

You can now use '/usr/local/bin/gmake' to build Rakudo.
After that, '/usr/local/bin/gmake test' will run some tests and
'/usr/local/bin/gmake install' will install Rakudo.
bastille% make                                                                 
'perl' -I'/home/morfent/Documents/perl6/rakudo/tools/lib' -I'/home/morfent/Documents/perl6/rakudo/3rdparty/nqp-configure/lib' '/home/morfent/Documents/perl6/rakudo/tools/build/check-nqp-version.pl' '/home/morfent/.perl6/bin/nqp-m'
'perl' -I'/home/morfent/Documents/perl6/rakudo/tools/lib' -I'/home/morfent/Documents/perl6/rakudo/3rdparty/nqp-configure/lib' '/home/morfent/Documents/perl6/rakudo/Configure.pl' --no-relocatable="1" --prefix="/home/morfent/.perl6" --backends=moar --expand main-version --out 'gen/moar/main-version.nqp'
Option no-relocatable does not take an argument
Configure.pl - Rakudo Configure

General Options:
    --help             Show this text
    --prefix=<path>    Install files in dir; also look for executables there
    --libdir=<path>    Install architecture-specific files in dir; Perl6 modules
                       included
    --no-relocatable
                       Create a perl6 with a fixed NQP and Perl6 home dir
                       instead of dynamically identifying it (On AIX MoarVM is
                       always built non-relocatable, since AIX misses
                       a necessary mechanism.)
    --sdkroot=<path>   When given, use for searching build tools here, e.g.
                       nqp, java, node etc.
    --sysroot=<path>   When given, use for searching runtime components here
    --backends=jvm,moar,js
                       Which backend(s) to use (or ALL for all of them)
    --gen-nqp[=branch]
                       Download, build, and install a copy of NQP before writing
                       the Makefile
    --gen-moar[=branch]
                       Download, build, and install a copy of MoarVM to use
                       before writing the Makefile
    --with-nqp=<path>
                       Provide path to already installed nqp
    --make-install     Install Rakudo after configuration is done
    --moar-option='--option=value'
                       Options to pass to MoarVM's Configure.pl
                       For example: --moar-option='--compiler=clang'
    --github-user=<user>
                       Fetch all repositories (rakudo, nqp, roast, MoarVM) from
                       this github user. Note that the user must have all
                       required repos forked from the originals.
    --rakudo-repo=<url>
    --nqp-repo=<url>
    --moar-repo=<url>
    --roast-repo=<url>
                       User-defined URL to fetch corresponding components
                       from. The URL will also be used to setup git push.
    --git-protocol={ssh,https,git}
                       Protocol used for cloning git repos
    --git-depth=<number>
                       Use the --git-depth option for git clone with parameter number
    --git-reference=<path>
                       Use --git-reference option to identify local path where git repositories are stored
                       For example: --git-reference=/home/user/repo/for_perl6
                       Folders 'nqp' and 'MoarVM' with corresponding git repos should be in for_perl6 folder
    --makefile-timing  Enable timing of individual makefile commands
    --no-clean         Skip cleanup before installation
    --ignore-errors    Ignore errors (such as the version of NQP)
    --expand=<template>
                       Expand template file. With this option Makefile is not
                       generated. The result is send to stdout unless --out
                       specified.
    --out=<file>       Filename to send output of --expand into.
    --set-var="config_variable=value"
                       Sets a config_variable to "value". Can be used multiple
                       times.

Please note that the --gen-moar and --gen-nqp options are there for convenience
only and will actually immediately - at Configure time - compile and install
moar and nqp respectively. They will live under the path given to --prefix,
unless other targeting options are used. To configure how MoarVM should be
compiled, use the --moar-option flag and view MoarVM's Configure.pl for more
information on its configuration options.

Configure.pl also reads options from 'config.default' in the current directory.
*** Error 1 in /home/morfent/Documents/perl6/rakudo (Makefile:534 'gen/moar/main-version.nqp')

vrurg · 2019-05-17T14:15:06Z

gmake won't be recommended if not found on PATH. The logic is to detect a make and then inform the user about the found one. Unfortunately, BSD makes doesn't identify themselves in their output making it impossible to make sure what exact make family we have at our hands. For this reason I choose the one I would be sure about.

vrurg · 2019-05-17T14:17:07Z

Thanks for the output! I know what to do about it. You say openBSD won't build relocatable binary no matter what? Maybe there is a workaround?

Kaiepi · 2019-05-17T14:19:03Z

Thanks for the output! I know what to do about it. You say openBSD won't build relocatable binary no matter what? Maybe there is a workaround?

The workaround is to call Configure.pl with --no-relocatable, then manually edit CONFIGURE in the makefile to contain --no-relocatable instead of --no-relocatable="1".

Edit: I'm dumb, see my next comment

patrickbkr · 2019-05-17T14:20:00Z

@vrurg: During my relocatable work me and Kaiepi already fought the relocatable on OpenBSD battle and lost. Hard. The best we can do at the moment is keep OpenBSD non-relocatable.

Kaiepi · 2019-05-17T14:20:29Z

I can't read. There is no workaround; $ORIGIN doesn't work on OpenBSD, making it impossible for relocatable executables to work.

ugexe · 2019-05-17T14:29:07Z

Unfortunately, BSD makes doesn't identify themselves in their output making it impossible to make sure what exact make family we have at our hands. For this reason I choose the one I would be sure about.

What about creating a temp file GNUmakefile and seeing if make runs it? Kind of sloppy, but seems like it would work.

Kaiepi · 2019-05-17T14:34:51Z

What about creating a temp file GNUmakefile and seeing if make runs it? Kind of sloppy, but seems like it would work.

A better approach would be running make -v. -v is unsupported on BSD make, while it is on GNU make.

vrurg · 2019-05-17T14:39:15Z

@ugexe BSDmakefile for that matter would be better.

@Kaiepi I'm coming from the times when it was possible to have like 4-5 different make flavors around. Wouldn't take proof by contradiction for granted.

In either case, I'll take care of no-relocatable option and will force-enable it for OpenBSD.

Make OpenBSD not relocatable by default. Improve building of command line options for Configure.pl rakudo/rakudo#2904 rakudo/rakudo#2904

vrurg · 2019-05-17T16:59:59Z

@Kaiepi you could try the above PRs while I'm awaiting for travis/appveyor tests passing.

vrurg · 2019-05-17T17:46:08Z

On my test OpenBSD everything works until the installation phase:

'/home/vrurg/rakudo/perl6-m' '/home/vrurg/rakudo/tools/build/install-core-dist.p6' '/home/vrurg/rakudo/install/share/perl6'
===SORRY!===
no such file or directory

Not sure if I'll have time today to look into it.

vrurg · 2019-05-18T03:46:03Z

@Kaiepi My lates changes from #2906 seemingly allows to install on OpenBSD. Have no time today for more testing.

Benabik · 2019-05-20T13:23:49Z

I'm hitting this same problem on macOS, which does use GNU make. Can confirm #2906 fixes the problem for me.

vrurg · 2019-05-20T14:25:26Z

I'm somewhat hesitate about merging #2906 yet. But will likely merge it later today as the problems I encountered are seemingly not related to the changes.

vrurg · 2019-05-20T22:34:19Z

@Kaiepi could you please confirm if the build is working for you now? My VM installation seems to be installing ok though failing spectests. But that's unlikely to be build-related problem. More like platform support issue.

UPD roast is now ok too.

Kaiepi · 2019-05-21T12:43:04Z

I can't get Configure.pl to run:

bastille% ./Configure.pl --gen-nqp --backends=moar --prefix=$HOME/.perl6
HEAD is now at 637207789 Bump Moar to get the latest goodies
Can't locate object method "make_option" via package "NQP::Config::Rakudo" at /home/morfent/Documents/perl6/rakudo/tools/lib/NQP/Config/Rakudo.pm line 429.
 at /home/morfent/Documents/perl6/rakudo/3rdparty/nqp-configure/lib/NQP/Config.pm line 34.
	NQP::Config::__ANON__("Can't locate object method \"make_option\" via package \"NQP::Co"...) called at /home/morfent/Documents/perl6/rakudo/tools/lib/NQP/Config/Rakudo.pm line 429
	NQP::Config::Rakudo::gen_nqp(NQP::Config::Rakudo=HASH(0x1ef02a8316d0)) called at ./Configure.pl line 98

vrurg · 2019-05-21T13:32:35Z

You need to git submodule update. Or, considering recent migration of nqp-configure submodule under perl6 ownership, you can either:

git submodule sync
git submodule update --init --recursive --remote

or re-clone rakudo from scratch.

Kaiepi · 2019-05-21T18:13:35Z

The build works.

Kaiepi · 2019-05-21T18:21:56Z

However one of the commits caused a regression:

bastille% perl6 --ll-exception -e 'say $*EXECUTABLE'
"--ll-exception".IO

$*EXECUTABLE and $*EXECUTABLE-NAME still need special cases for OpenBSD.

vrurg · 2019-05-21T19:06:58Z

Ah, different handling is perhaps needed for different scenarios. I'll have a look into what can be done to this.

_{Sent with GitHawk}

sabazk · 2019-05-21T21:11:03Z

I'm on OpenBSD 6.5, rakudo builds with clang from ports on my box, but when I run the executable(install/bin/perl6) I get : Unhandled exception: Malformed UTF-8 at line 1 col 85
however the perl6 binary in the root rakudo directory can be run with no problems (./perl6)

vrurg · 2019-05-22T17:53:18Z

@Kaiepi I finally got a bit of time to look into it. Seems like it's to be fixed in moar because if it can't determine correct executable then nqp::execname() must return an empty one.

@javan-z I will try to build with llvm from ports, but it doesn't look like it belongs here. A separate ticket would be needed.

vrurg · 2019-05-22T18:04:01Z

@javan-z My build has completed with clang from ports and everything seems to be ok.

vrurg · 2019-05-23T00:29:17Z

@Kaiepi I merged #2915. It fixes the problem.

sabazk · 2019-05-23T18:56:02Z

Works beautifully, Thanks

vrurg · 2019-06-01T01:03:44Z

Closed via #2915

Kaiepi · 2019-06-05T18:46:18Z

The build works fine, but instead of $*EXECUTABLE being the absolute path to the perl6 binary, I get this:

bastille% perl6 -e 'say $*EXECUTABLE'
"perl6".IO
bastille% perl6 --ll-exception -e 'say $*EXECUTABLE'
"perl6".IO

This makes it impossible to run any modules.

vrurg · 2019-06-05T21:42:34Z

Did you expect $*EXECUTABLE to be a string? Because this is what I get on macOS with 2019.03:

"/Users/vrurg/.rakudobrew/bin/../versions/moar-2019.03.1/install/bin/perl6".IO

With regard to the full path – I didn't find a way to get it. It's OpenBSD, after all. 😉

ugexe · 2019-06-05T21:49:50Z

The problem is $*EXECUTABLE.absolute should work. "perl6".IO.absolute will only ever be right when a perl6 binary resides in cwd.

rakudo/src/core/CompUnit/PrecompilationRepository.pm6

Line 252 in 32d480a

my $perl6 = $*EXECUTABLE.absolute

If the above did not call .absolute then when manually running /home/alt/perl6 (not in PATH) it might precompile things with the perl6 in my path at /home/ugexe/perl6.

vrurg · 2019-06-06T03:09:04Z

.absolute doesn't provide the right path. The problem is that OpenBSD doesn't let a process to know the full path of its executable. And as far as I know there is no way to resolve this. I found a StackOverflow topic where they cite Theo de Raadt saying: Quite expensive, for such a small need.

For now I don't see a good solution for this. Perhaps the least painful one would be macro expansion of runner's C source and manual insertion of the full path as defined in the Makefile – similar to how scripted runners are being generated.

patrickbkr · 2019-06-06T08:24:20Z

I'll have a look at this. I think I know what's to do.

patrickbkr · 2019-06-06T08:28:43Z

I think we might be able to remove all the OpenBSD special casing when we set the executable path statically in the non-relocatable case. That's basically what @vrurg proposed.
@vrurg Are you ok with me taking this ticket? Would you prefer another -Dvar=value C compiler variable to pass the executable name into the C runner or turning it into a template?

vrurg · 2019-06-06T13:26:36Z

@PatZim you're welcome! It's really closer to your specialization. -D is really much better solution than templating in this case.

In the non-relocatable case it's not necessary to determine the executable path at runtime. We can just as well compile it in. Doing so solves the problem of OpenBSD where it's currently not technically possible to determine the path of an executable from within. Fixes rakudo#2904

In the non-relocatable case it's not necessary to determine the executable path at runtime. We can just as well compile it in. Doing so solves the problem of OpenBSD where it's currently not technically possible to determine the path of an executable from within. Fixes rakudo#2904 Also some cleanup of the C runner.

vrurg self-assigned this May 17, 2019

vrurg added a commit to Raku/nqp-configure that referenced this issue May 17, 2019

Fixing OpenBSD build

513b9bc

Make OpenBSD not relocatable by default. Improve building of command line options for Configure.pl rakudo/rakudo#2904 rakudo/rakudo#2904

This was referenced May 17, 2019

Support for the latest changes in nqp-configure Raku/nqp#539

Merged

Support for latest changes in nqp-configure #2906

Merged

vrurg mentioned this issue May 23, 2019

Resolve a problem with executable name on OpenBSD #2915

Merged

vrurg closed this as completed May 25, 2019

patrickbkr assigned patrickbkr and unassigned vrurg Jun 6, 2019

patrickbkr reopened this Jun 6, 2019

patrickbkr mentioned this issue Jun 7, 2019

Use a static exec path with non-reloc builds #2963

Merged

patrickbkr closed this as completed in #2963 Jun 12, 2019

Rakudo no longer builds properly on OpenBSD #2904

Rakudo no longer builds properly on OpenBSD #2904

Comments

Kaiepi commented May 17, 2019

The Problem

Expected Behavior

Actual Behavior

Steps to Reproduce

Environment

vrurg commented May 17, 2019

vrurg commented May 17, 2019

Kaiepi commented May 17, 2019

vrurg commented May 17, 2019

vrurg commented May 17, 2019

Kaiepi commented May 17, 2019 • edited

patrickbkr commented May 17, 2019

Kaiepi commented May 17, 2019

ugexe commented May 17, 2019

Kaiepi commented May 17, 2019

vrurg commented May 17, 2019

vrurg commented May 17, 2019

vrurg commented May 17, 2019

vrurg commented May 18, 2019

Benabik commented May 20, 2019

vrurg commented May 20, 2019

vrurg commented May 20, 2019 • edited

Kaiepi commented May 21, 2019

vrurg commented May 21, 2019

Kaiepi commented May 21, 2019

Kaiepi commented May 21, 2019

vrurg commented May 21, 2019

sabazk commented May 21, 2019 • edited

vrurg commented May 22, 2019

vrurg commented May 22, 2019

vrurg commented May 23, 2019 • edited

sabazk commented May 23, 2019

vrurg commented Jun 1, 2019

Kaiepi commented Jun 5, 2019

vrurg commented Jun 5, 2019

ugexe commented Jun 5, 2019 • edited

vrurg commented Jun 6, 2019

patrickbkr commented Jun 6, 2019

patrickbkr commented Jun 6, 2019

vrurg commented Jun 6, 2019

Kaiepi commented May 17, 2019 •

edited

vrurg commented May 20, 2019 •

edited

sabazk commented May 21, 2019 •

edited

vrurg commented May 23, 2019 •

edited

ugexe commented Jun 5, 2019 •

edited