gnulib: Fix path to perl in prefix-gnulib-mk #274161

afh · 2023-12-14T06:57:26Z

Description of changes

While building a custom groff using nixpkgs I came across the following error:

/nix/store/cic1hyg92ggj4swjp37phggvdacq2kz6-source/build-aux/prefix-gnulib-mk: line 26: exec: perl: not found

Inspecing prefix-gnulib-mk perl is used in line 26. Hence replacing perl with a proper path to perl in nix seems necessary.

❓ Please let me know if you believe this PR should rather be applied to master than staging. Since it

Things done

Add a 👍 reaction to pull requests you find important.

SuperSandro2000 · 2023-12-18T22:37:49Z

Did we before rely on perl being in PATH or did we just never hit that code path?

Also this should go to master, since it rebuilds only ~10 or so packages.

afh · 2023-12-19T12:09:02Z

Thank you for your feedback, @SuperSandro2000, this PR's merge base has now been changed to master.

Unfortunately I cannot answer that question with confidence, I assume that we never hit that code path. I only came across it while trying to manually build groff from within a nix develop shell.

SuperSandro2000 · 2023-12-19T13:03:03Z

I think if we can keep relying on PATH we can keep the build closures of all other packages smaller which would be need but perl is probably already somewhere in there

afh · 2023-12-19T14:22:11Z

Happy to look for alternative solutions, @SuperSandro2000 and maybe you have time and interest to partake in finding them.

Inspecting gnulib's build-aux/prefix-gnulib-mk:26 it runs eval 'exec perl -wSx "$0" "$@"' for some reason unclear to me the following error appears while trying to build groff from within a nix develop shell: /nix/store/cic1hyg92ggj4swjp37phggvdacq2kz6-source/build-aux/prefix-gnulib-mk: line 26: exec: perl: not found.

Inside and outside that build environment perl is /usr/bin/perl. Is there anything from nixpkgs that comes to your mind, that would change PATH in a way that it is no longer found by prefix-gnulib-mk?

Is there a way in nixpkgs with which perl could be substituted for a valid absolute path to the perl executable on the current system (for me on macos /usr/bin/perl)?

What are some of the things you'd try?

SuperSandro2000 · 2023-12-30T01:26:42Z

Is there anything from nixpkgs that comes to your mind, that would change PATH in a way that it is no longer found by prefix-gnulib-mk?

In the sandbox it is cleared.

Is there a way in nixpkgs with which perl could be substituted for a valid absolute path to the perl executable on the current system (for me on macos /usr/bin/perl)?

We should probably replace the shebang on line 1 with a perl shebang to avoid this hackery.

pkgs/development/tools/gnulib/default.nix

afh · 2024-01-03T18:50:42Z

Thanks for the feedback and suggestion, @SuperSandro2000.
Is there another way you can think of that would help with keeping the build closures smaller as you requested earlier?

afh · 2024-01-11T07:25:07Z

I'd like to post this PR in the "PRs already reviewed" channel and kindly request re-review from you, @SuperSandro2000, as you've taken at look at this previously.

pbsds · 2024-01-11T21:54:57Z

Result of nixpkgs-review pr 274161 run on x86_64-linux 1

32 packages built:

gnu-pw-mgr
gnulib
idutils
lbzip2
lua51Packages.lrexlib-gnu
luaPackages.lrexlib-gnu (lua52Packages.lrexlib-gnu)
lua53Packages.lrexlib-gnu
lua54Packages.lrexlib-gnu
luajitPackages.lrexlib-gnu
python310Packages.condaInstallHook
python310Packages.torchWithRocm
python310Packages.torchWithRocm.cxxdev
python310Packages.torchWithRocm.dev
python310Packages.torchWithRocm.dist
python310Packages.torchWithRocm.lib
python311Packages.condaInstallHook
python311Packages.torchWithRocm
python311Packages.torchWithRocm.cxxdev
python311Packages.torchWithRocm.dev
python311Packages.torchWithRocm.dist
python311Packages.torchWithRocm.lib
rocmPackages.migraphx (rocmPackages_5.migraphx)
rocmPackages.miopen (rocmPackages.miopen-hip ,rocmPackages_5.miopen ,rocmPackages_5.miopen-hip)
rocmPackages.miopen-opencl (rocmPackages_5.miopen-opencl)
rocmPackages.mivisionx (rocmPackages.mivisionx-hip ,rocmPackages_5.mivisionx ,rocmPackages_5.mivisionx-hip)
rocmPackages.mivisionx-cpu (rocmPackages_5.mivisionx-cpu)
rocmPackages.mivisionx-opencl (rocmPackages_5.mivisionx-opencl)
surf-display
wget2
wget2.dev
wget2.lib
xprintidle-ng

Shebang is properly patched as intended, and the executable seems to work. LGTM

This seems to produce the only in-store reference to perl, which means the package previously was non-hermetic. There are other occurrences of files which tries to exec into perl from sh, if this is the direction you want to go, please patch them as well

build-aux/useless-if-before-free
build-aux/update-copyright
build-aux/gitlog-to-changelog
build-aux/announce-gen

pbsds · 2024-01-14T18:37:53Z

Result of nixpkgs-review pr 274161 run on x86_64-linux 1

5 packages failed to build:

idutils

succeeds on hydra

rocmPackages.migraphx (rocmPackages_5.migraphx)

fails on hydra

rocmPackages.mivisionx (rocmPackages.mivisionx-hip ,rocmPackages_5.mivisionx ,rocmPackages_5.mivisionx-hip)

fails on hydra

rocmPackages.mivisionx-cpu (rocmPackages_5.mivisionx-cpu)

fails on hydra

rocmPackages.mivisionx-opencl (rocmPackages_5.mivisionx-opencl)

fails on hydra

27 packages built:

gnu-pw-mgr
gnulib
lbzip2
lua51Packages.lrexlib-gnu
luaPackages.lrexlib-gnu (lua52Packages.lrexlib-gnu)
lua53Packages.lrexlib-gnu
lua54Packages.lrexlib-gnu
luajitPackages.lrexlib-gnu
python310Packages.condaInstallHook
python310Packages.torchWithRocm
python310Packages.torchWithRocm.cxxdev
python310Packages.torchWithRocm.dev
python310Packages.torchWithRocm.dist
python310Packages.torchWithRocm.lib
python311Packages.condaInstallHook
python311Packages.torchWithRocm
python311Packages.torchWithRocm.cxxdev
python311Packages.torchWithRocm.dev
python311Packages.torchWithRocm.dist
python311Packages.torchWithRocm.lib
rocmPackages.miopen (rocmPackages.miopen-hip ,rocmPackages_5.miopen ,rocmPackages_5.miopen-hip)
rocmPackages.miopen-opencl (rocmPackages_5.miopen-opencl)
surf-display
wget2
wget2.dev
wget2.lib
xprintidle-ng

from idutils build log:

FAIL: test-update-copyright.sh

idutils seems to perform many gnulib tests, does it assume a specific/vendored gnulib @gfrascadorio?

EDIT: the tests seems to test build-aux/update-copyright, which now spits out the following warnings:

$ results/gnulib/build-aux/update-copyright
Use of uninitialized value $_ in pattern match (m//) at results/gnulib/build-aux/update-copyright line 171.
Use of uninitialized value $_ in pattern match (m//) at results/gnulib/build-aux/update-copyright line 177.
Use of uninitialized value $ARGV in concatenation (.) or string at results/gnulib/build-aux/update-copyright line 293.
: warning: copyright statement not found

afh · 2024-06-02T04:30:23Z

Friendly ping to @anthonyroussel, @SuperSandro2000, and @pbsds on this to have another look at the changes proposed in this PR 🙂

pkgs/development/tools/gnulib/default.nix

afh · 2024-06-03T07:31:26Z

Thanks for the scrutinising review and and helpful comments, @pbsds and @SuperSandro2000, very much appreciated!

For good measure I took the liberty to use ${lib.getBin perl} instead of just ${perl} for the replacement, would you agree that that's a good call?

pbsds · 2024-06-05T18:32:19Z

lib.getBin makes sense, but lib.getExe might be the better option.

Result of nixpkgs-review pr 274161 run on x86_64-linux 1

8 packages marked as broken and skipped:

rocmPackages.migraphx
rocmPackages.mivisionx
rocmPackages.mivisionx-cpu
rocmPackages.mivisionx-hip
rocmPackages_6.migraphx
rocmPackages_6.mivisionx
rocmPackages_6.mivisionx-cpu
rocmPackages_6.mivisionx-hip

14 packages failed to build:

idutils
python311Packages.torchWithRocm
python311Packages.torchWithRocm.cxxdev
python311Packages.torchWithRocm.dev
python311Packages.torchWithRocm.dist
python311Packages.torchWithRocm.lib
python312Packages.torchWithRocm
python312Packages.torchWithRocm.cxxdev
python312Packages.torchWithRocm.dev
python312Packages.torchWithRocm.dist
python312Packages.torchWithRocm.lib
rocmPackages_5.mivisionx (rocmPackages_5.mivisionx-hip)
rocmPackages_5.mivisionx-cpu
rocmPackages_5.mivisionx-opencl

20 packages built:

gnu-pw-mgr
gnulib
lbzip2
lua51Packages.lrexlib-gnu
luaPackages.lrexlib-gnu (lua52Packages.lrexlib-gnu)
lua53Packages.lrexlib-gnu
lua54Packages.lrexlib-gnu
luajitPackages.lrexlib-gnu
python311Packages.condaInstallHook
python312Packages.condaInstallHook
rocmPackages.miopen (rocmPackages.miopen-hip ,rocmPackages_6.miopen ,rocmPackages_6.miopen-hip)
rocmPackages_5.migraphx
rocmPackages_5.miopen (rocmPackages_5.miopen-hip)
rocmPackages_5.miopen-opencl
surf-display
wget2
wget2.dev
wget2.lib
xprintidle-ng
zluda

idutils still fails.

$ results/gnulib/build-aux/update-copyright
Use of uninitialized value $_ in pattern match (m//) at results/gnulib/build-aux/update-copyright line 171.
Use of uninitialized value $_ in pattern match (m//) at results/gnulib/build-aux/update-copyright line 177.
Use of uninitialized value $ARGV in concatenation (.) or string at results/gnulib/build-aux/update-copyright line 293.
: warning: copyright statement not found

the other failures also fail on hydra

afh · 2024-06-05T18:42:21Z

Thanks, @pbsds, I'll look into it.

What's the difference between lib.getBin and lib.getExe (feel free to point me to relevant information should be explanation be too much effort).

afh · 2024-06-05T18:55:14Z

One cause for the build failures could be that the update-copyright script is not properly invoked from idutils as it appears odd to me that $ARGV is empty as well as $_ the default input or search space variable… 🤔

Here are links to gnulib's update-copyright (respecting the src.rev used in nixpkgs):

pbsds · 2024-06-05T19:44:31Z

It expands to ${lib.getBin drv}/bin/${drv.meta.mainProgram}

$ nix repl '<nixpkgs>'
Added 19880 variables.
nix-repl> lib.getBin hello
«derivation /nix/store/i6mny38n5j529m24sgnwl7jipishjmzb-hello-2.12.1.drv»
nix-repl> lib.getExe hello 
"/nix/store/fchfx8ykh12d0f8bq0yi3k80mcz7n250-hello-2.12.1/bin/hello"

I tried running the one currently in master, and it waits on stdin

$ $(nix build nixpkgs#gnulib --print-out-paths)/build-aux/update-copyright
-i used with no filenames on the command line, reading from STDIN.
^D
-: warning: copyright statement not found

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>

afh · 2024-06-23T18:46:58Z

Result of nixpkgs-review pr 274161 run on aarch64-darwin 1

1 package marked as broken and skipped:

zluda

8 packages built:

gnulib
idutils
lbzip2
lua51Packages.lrexlib-gnu
luaPackages.lrexlib-gnu (lua52Packages.lrexlib-gnu)
lua53Packages.lrexlib-gnu
lua54Packages.lrexlib-gnu
luajitPackages.lrexlib-gnu

afh · 2024-06-23T18:51:13Z

Apologies for the silence on this issue, folks. My attention was needed elsewhere. On the plus side: I now also have access to a x86_64-linux system for testing and can confirm that building the idutils package from this branch fails the update-copyright test.

What's curious is that idutils from this branch builds and passes the tests on aarch64-darwin. Any one have an idea what could cause the issue on linux?

afh · 2024-06-23T19:04:14Z

Maybe it's alright to skip/disable the update-copyright test since:

It is important to have self tests for programs like that, but it's not as important that they run properly on non-development platforms.
— https://lists.nongnu.org/archive/html/bug-gnulib/2011-11/msg00320.html

What do people think?

afh · 2024-06-23T19:54:14Z

@pbsds I went ahead and disable the failing test on Linux. Would you have time and interest in having another look at giving it a try?

nixos-discourse · 2024-06-23T19:56:17Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/4139

afh · 2024-06-27T12:45:09Z

Friendly ping to the current reviewers on this one 🙂
@anthonyroussel @pbsds @SuperSandro2000 @gfrascadorio

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 11-100 labels Dec 14, 2023

afh requested review from SuperSandro2000 and anthonyroussel December 15, 2023 11:31

afh force-pushed the gnulib-subst-perl branch from 6b42951 to 7436efb Compare December 19, 2023 12:04

afh changed the base branch from staging to master December 19, 2023 12:05

afh force-pushed the gnulib-subst-perl branch from 7436efb to 6185761 Compare December 19, 2023 12:06

SuperSandro2000 reviewed Dec 30, 2023

View reviewed changes

pkgs/development/tools/gnulib/default.nix Outdated Show resolved Hide resolved

afh force-pushed the gnulib-subst-perl branch from d94c483 to 8e9e83e Compare January 11, 2024 07:19

afh requested a review from SuperSandro2000 January 11, 2024 07:25

afh force-pushed the gnulib-subst-perl branch from 8e9e83e to 54f41cc Compare January 11, 2024 23:58

afh marked this pull request as draft June 1, 2024 15:57

afh force-pushed the gnulib-subst-perl branch from 54f41cc to 6c6d954 Compare June 1, 2024 15:57

afh marked this pull request as ready for review June 1, 2024 18:16

SuperSandro2000 reviewed Jun 2, 2024

View reviewed changes

pkgs/development/tools/gnulib/default.nix Outdated Show resolved Hide resolved

afh force-pushed the gnulib-subst-perl branch 2 times, most recently from d90301e to f00622e Compare June 3, 2024 07:29

afh requested review from pbsds and SuperSandro2000 June 3, 2024 07:29

afh force-pushed the gnulib-subst-perl branch from f00622e to 47ece0d Compare June 5, 2024 20:07

gnulib: Fix path to perl in prefix-gnulib-mk

7c306a3

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>

afh force-pushed the gnulib-subst-perl branch from 47ece0d to 7c306a3 Compare June 23, 2024 17:04

idutils: disable failing tests on linux

be47faa

ofborg bot requested a review from gfrascadorio June 23, 2024 20:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gnulib: Fix path to perl in prefix-gnulib-mk #274161

gnulib: Fix path to perl in prefix-gnulib-mk #274161

afh commented Dec 14, 2023

SuperSandro2000 commented Dec 18, 2023 •

edited

Loading

afh commented Dec 19, 2023

SuperSandro2000 commented Dec 19, 2023

afh commented Dec 19, 2023

SuperSandro2000 commented Dec 30, 2023

afh commented Jan 3, 2024

afh commented Jan 11, 2024

pbsds commented Jan 11, 2024

pbsds commented Jan 14, 2024 •

edited

Loading

afh commented Jun 2, 2024

afh commented Jun 3, 2024

pbsds commented Jun 5, 2024 •

edited

Loading

afh commented Jun 5, 2024

afh commented Jun 5, 2024

pbsds commented Jun 5, 2024

afh commented Jun 23, 2024

afh commented Jun 23, 2024

afh commented Jun 23, 2024

afh commented Jun 23, 2024

nixos-discourse commented Jun 23, 2024

afh commented Jun 27, 2024

gnulib: Fix path to perl in prefix-gnulib-mk #274161

Are you sure you want to change the base?

gnulib: Fix path to perl in prefix-gnulib-mk #274161

Conversation

afh commented Dec 14, 2023

Description of changes

Things done

SuperSandro2000 commented Dec 18, 2023 • edited Loading

afh commented Dec 19, 2023

SuperSandro2000 commented Dec 19, 2023

afh commented Dec 19, 2023

SuperSandro2000 commented Dec 30, 2023

afh commented Jan 3, 2024

afh commented Jan 11, 2024

pbsds commented Jan 11, 2024

pbsds commented Jan 14, 2024 • edited Loading

afh commented Jun 2, 2024

afh commented Jun 3, 2024

pbsds commented Jun 5, 2024 • edited Loading

afh commented Jun 5, 2024

afh commented Jun 5, 2024

pbsds commented Jun 5, 2024

afh commented Jun 23, 2024

afh commented Jun 23, 2024

afh commented Jun 23, 2024

afh commented Jun 23, 2024

nixos-discourse commented Jun 23, 2024

afh commented Jun 27, 2024

SuperSandro2000 commented Dec 18, 2023 •

edited

Loading

pbsds commented Jan 14, 2024 •

edited

Loading

pbsds commented Jun 5, 2024 •

edited

Loading