Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: expand explanation of patchShebangs hook #121015

Merged
merged 9 commits into from
Jul 13, 2022
77 changes: 70 additions & 7 deletions doc/stdenv/stdenv.chapter.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ where the builder can do anything it wants, but typically starts with
source $stdenv/setup
```

to let `stdenv` set up the environment (e.g., process the `buildInputs`). If you want, you can still use `stdenv`’s generic builder:
to let `stdenv` set up the environment (e.g. by resetting `PATH` and populating it from build inputs). If you want, you can still use `stdenv`’s generic builder:

```bash
source $stdenv/setup
Expand Down Expand Up @@ -644,12 +644,12 @@ Hook executed at the end of the install phase.

### The fixup phase {#ssec-fixup-phase}

The fixup phase performs some (Nix-specific) post-processing actions on the files installed under `$out` by the install phase. The default `fixupPhase` does the following:
The fixup phase performs (Nix-specific) post-processing actions on the files installed under `$out` by the install phase. The default `fixupPhase` does the following:

- It moves the `man/`, `doc/` and `info/` subdirectories of `$out` to `share/`.
- It strips libraries and executables of debug information.
- On Linux, it applies the `patchelf` command to ELF executables and libraries to remove unused directories from the `RPATH` in order to prevent unnecessary runtime dependencies.
- It rewrites the interpreter paths of shell scripts to paths found in `PATH`. E.g., `/usr/bin/perl` will be rewritten to `/nix/store/some-perl/bin/perl` found in `PATH`.
- It rewrites the interpreter paths of shell scripts to paths found in `PATH`. E.g., `/usr/bin/perl` will be rewritten to `/nix/store/some-perl/bin/perl` found in `PATH`. See [](#patch-shebangs.sh) for details.

#### Variables controlling the fixup phase {#variables-controlling-the-fixup-phase}

Expand Down Expand Up @@ -695,7 +695,7 @@ If set, the `patchelf` command is not used to remove unnecessary `RPATH` entries

##### `dontPatchShebangs` {#var-stdenv-dontPatchShebangs}

If set, scripts starting with `#!` do not have their interpreter paths rewritten to paths in the Nix store.
If set, scripts starting with `#!` do not have their interpreter paths rewritten to paths in the Nix store. See [](#patch-shebangs.sh) on how patching shebangs works.

##### `dontPruneLibtoolFiles` {#var-stdenv-dontPruneLibtoolFiles}

Expand Down Expand Up @@ -929,7 +929,7 @@ addEnvHooks "$hostOffset" myBashFunction

The *existence* of setups hooks has long been documented and packages inside Nixpkgs are free to use this mechanism. Other packages, however, should not rely on these mechanisms not changing between Nixpkgs versions. Because of the existing issues with this system, there’s little benefit from mandating it be stable for any period of time.

First, let’s cover some setup hooks that are part of Nixpkgs default stdenv. This means that they are run for every package built using `stdenv.mkDerivation`. Some of these are platform specific, so they may run on Linux but not Darwin or vice-versa.
First, let’s cover some setup hooks that are part of Nixpkgs default `stdenv`. This means that they are run for every package built using `stdenv.mkDerivation` or when using a custom builder that has `source $stdenv/setup`. Some of these are platform specific, so they may run on Linux but not Darwin or vice-versa.

### `move-docs.sh` {#move-docs.sh}

Expand All @@ -945,7 +945,70 @@ This runs the strip command on installed binaries and libraries. This removes un

### `patch-shebangs.sh` {#patch-shebangs.sh}

This setup hook patches installed scripts to use the full path to the shebang interpreter. A shebang interpreter is the first commented line of a script telling the operating system which program will run the script (e.g `#!/bin/bash`). In Nix, we want an exact path to that interpreter to be used. This often replaces `/bin/sh` with a path in the Nix store.
This setup hook patches installed scripts to add Nix store paths to their shebang interpreter as found in the build environment. The [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) line tells a Unix-like operating system which interpreter to use to execute the script's contents.

::: note
fricklerhandwerk marked this conversation as resolved.
Show resolved Hide resolved
The [generic builder][generic-builder] populates `PATH` from inputs of the derivation.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe link to the place that actually does that instead, since people might click the link and be confused.

:::

[generic-builder]: https://github.com/NixOS/nixpkgs/blob/19d4f7dc485f74109bd66ef74231285ff797a823/pkgs/stdenv/generic/builder.sh

#### Invocation
fricklerhandwerk marked this conversation as resolved.
Show resolved Hide resolved

Multiple paths can be specified.

```
patchShebangs [--build | --host] PATH...
```

#### Flags

`--build`
: Look up commands available at build time

`--host`
: Look up commands available at run time

#### Examples
fricklerhandwerk marked this conversation as resolved.
Show resolved Hide resolved
fricklerhandwerk marked this conversation as resolved.
Show resolved Hide resolved

```sh
patchShebangs --host /nix/store/<hash>-hello-1.0/bin
```

```sh
patchShebangs --build configure
```

`#!/bin/sh` will be rewritten to `#!/nix/store/<hash>-some-bash/bin/sh`.

`#!/usr/bin/env` gets special treatment: `#!/usr/bin/env python` is rewritten to `/nix/store/<hash>/bin/python`.

Interpreter paths that point to a valid Nix store location are not changed.

::: note
A script file must be marked as executable, otherwise it will not be
considered. [Trivial Builders](#chap-trivial-builders) do this automatically where appropriate.
fricklerhandwerk marked this conversation as resolved.
Show resolved Hide resolved
:::

This mechanism ensures that the interpreter for a given script is always found and is exactly the one specified by the build.

It can be disabled by setting [`dontPatchShebangs`](#var-stdenv-dontPatchShebangs):

```nix
stdenv.mkDerivation {
# ...
dontPatchShebangs = true;
# ...
}
```

The file [`patch-shebangs.sh`][patch-shebangs.sh] defines the [`patchShebangs`][patchShebangs] function. It is used to implement [`patchShebangsAuto`][patchShebangsAuto], the [setup hook](#ssec-setup-hooks) that is registered to run during the [fixup phase](#ssec-fixup-phase) by default.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe move this to the beginning to serve as an overview.


If you need to run `patchShebangs` at build time, it must be called explicitly within [one of the build phases](#sec-stdenv-phases).

[patch-shebangs.sh]: https://github.com/NixOS/nixpkgs/blob/19d4f7dc485f74109bd66ef74231285ff797a823/pkgs/build-support/setup-hooks/patch-shebangs.sh
[patchShebangs]: https://github.com/NixOS/nixpkgs/blob/19d4f7dc485f74109bd66ef74231285ff797a823/pkgs/build-support/setup-hooks/patch-shebangs.sh#L24-L105
[patchShebangsAuto]: https://github.com/NixOS/nixpkgs/blob/19d4f7dc485f74109bd66ef74231285ff797a823/pkgs/build-support/setup-hooks/patch-shebangs.sh#L107-L119
fricklerhandwerk marked this conversation as resolved.
Show resolved Hide resolved

### `audit-tmpdir.sh` {#audit-tmpdir.sh}

Expand Down Expand Up @@ -1262,7 +1325,7 @@ If the libraries lack `-fPIE`, you will get the error `recompile with -fPIE`.

[^footnote-stdenv-ignored-build-platform]: The build platform is ignored because it is a mere implementation detail of the package satisfying the dependency: As a general programming principle, dependencies are always *specified* as interfaces, not concrete implementation.
[^footnote-stdenv-native-dependencies-in-path]: Currently, this means for native builds all dependencies are put on the `PATH`. But in the future that may not be the case for sake of matching cross: the platforms would be assumed to be unique for native and cross builds alike, so only the `depsBuild*` and `nativeBuildInputs` would be added to the `PATH`.
[^footnote-stdenv-propagated-dependencies]: Nix itself already takes a package’s transitive dependencies into account, but this propagation ensures nixpkgs-specific infrastructure like setup hooks (mentioned above) also are run as if the propagated dependency.
[^footnote-stdenv-propagated-dependencies]: Nix itself already takes a package’s transitive dependencies into account, but this propagation ensures nixpkgs-specific infrastructure like [setup hooks](#ssec-setup-hooks) also are run as if it were a propagated dependency.
[^footnote-stdenv-find-inputs-location]: The `findInputs` function, currently residing in `pkgs/stdenv/generic/setup.sh`, implements the propagation logic.
[^footnote-stdenv-sys-lib-search-path]: It clears the `sys_lib_*search_path` variables in the Libtool script to prevent Libtool from using libraries in `/usr/lib` and such.
[^footnote-stdenv-build-time-guessing-impurity]: Eventually these will be passed building natively as well, to improve determinism: build-time guessing, as is done today, is a risk of impurity.
Expand Down