Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Document "Import From Derivation" #7332

Merged
merged 15 commits into from
Sep 26, 2023
Merged

Document "Import From Derivation" #7332

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
7e58f11
document "Import From Derivation"
fricklerhandwerk Nov 18, 2022
fd93ec4
fixup wording
fricklerhandwerk Jul 13, 2023
048e0bb
fix links
fricklerhandwerk Jul 14, 2023
eec8998
add additional example
fricklerhandwerk Jul 17, 2023
2f3dfcb
clarify wording
fricklerhandwerk Jul 17, 2023
9f25018
clarify diagram label
fricklerhandwerk Jul 17, 2023
dadb3b2
clarify purpose of ifd option
fricklerhandwerk Jul 17, 2023
ffd9f7a
make example work without copying builder to the store
fricklerhandwerk Jul 17, 2023
3951a2b
explain `expr` sample
fricklerhandwerk Jul 19, 2023
700c8e1
go into more detail with explanations
fricklerhandwerk Jul 20, 2023
7e2b9a4
add more performance considerations
fricklerhandwerk Jul 20, 2023
b755328
reword explanation of example
fricklerhandwerk Jul 20, 2023
16c7902
be more precise about sequentiality and parallelism
fricklerhandwerk Sep 26, 2023
1b97893
avoid "output path"
fricklerhandwerk Sep 26, 2023
a05b62f
move definition to the beginning
fricklerhandwerk Sep 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/manual/src/SUMMARY.md.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
- [Operators](language/operators.md)
- [Derivations](language/derivations.md)
- [Advanced Attributes](language/advanced-attributes.md)
- [Import From Derivation](language/import-from-derivation.md)
fricklerhandwerk marked this conversation as resolved.
Show resolved Hide resolved
- [Built-in Constants](language/builtin-constants.md)
- [Built-in Functions](language/builtins.md)
- [Advanced Topics](advanced-topics/advanced-topics.md)
Expand Down
5 changes: 4 additions & 1 deletion doc/manual/src/glossary.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,12 +105,15 @@

- [store object]{#gloss-store-object}


A store object consists of a [file system object], [reference]s to other store objects, and other metadata.
It can be referred to by a [store path].

[store object]: #gloss-store-object

- [IFD]{#gloss-ifd}

[Import From Derivation](./language/import-from-derivation.md)

- [input-addressed store object]{#gloss-input-addressed-store-object}

A store object produced by building a
Expand Down
139 changes: 139 additions & 0 deletions doc/manual/src/language/import-from-derivation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
# Import From Derivation
fricklerhandwerk marked this conversation as resolved.
Show resolved Hide resolved

The value of a Nix expression can depend on the contents of a [store object](@docroot@/glossary.md#gloss-store-object).

Passing an expression `expr` that evaluates to a [store path](@docroot@/glossary.md#gloss-store-path) to any built-in function which reads from the filesystem constitutes Import From Derivation (IFD):

- [`import`](./builtins.md#builtins-import)` expr`
- [`builtins.readFile`](./builtins.md#builtins-readFile)` expr`
fricklerhandwerk marked this conversation as resolved.
Show resolved Hide resolved
- [`builtins.readFileType`](./builtins.md#builtins-readFileType)` expr`
- [`builtins.readDir`](./builtins.md#builtins-readDir)` expr`
- [`builtins.pathExists`](./builtins.md#builtins-pathExists)` expr`
- [`builtins.filterSource`](./builtins.md#builtins-filterSource)` f expr`
- [`builtins.path`](./builtins.md#builtins-path)` { path = expr; }`
- [`builtins.hashFile`](./builtins.md#builtins-hashFile)` t expr`
- `builtins.scopedImport x drv`

When the store path needs to be accessed, evaluation will be paused, the corresponding store object [realised], and then evaluation resumed.

[realised]: @docroot@/glossary.md#gloss-realise

This has performance implications:
Evaluation can only finish when all required store objects are realised.
Since the Nix language evaluator is sequential, it only finds store paths to read from one at a time.
While realisation is always parallel, in this case it cannot be done for all required store paths at once, and is therefore much slower than otherwise.

Realising store objects during evaluation can be disabled by setting [`allow-import-from-derivation`](../command-ref/conf-file.md#conf-allow-import-from-derivation) to `false`.
Without IFD it is ensured that evaluation is complete and Nix can produce a build plan before starting any realisation.
fricklerhandwerk marked this conversation as resolved.
Show resolved Hide resolved

## Example

In the following Nix expression, the inner derivation `drv` produces a file with contents `hello`.

```nix
# IFD.nix
let
drv = derivation {
name = "hello";
builder = "/bin/sh";
args = [ "-c" "echo -n hello > $out" ];
system = builtins.currentSystem;
};
in "${builtins.readFile drv} world"
```

```shellSession
nix-instantiate IFD.nix --eval --read-write-mode
```

```
building '/nix/store/348q1cal6sdgfxs8zqi9v8llrsn4kqkq-hello.drv'...
"hello world"
```

The contents of the derivation's output have to be [realised] before they can be read with [`readFile`](./builtins.md#builtins-readFile).
Only then evaluation can continue to produce the final result.

## Illustration

As a first approximation, the following data flow graph shows how evaluation and building are interleaved, if the value of a Nix expression depends on realising a [store object].
Boxes are data structures, arrow labels are transformations.

```
+----------------------+ +------------------------+
| Nix evaluator | | Nix store |
| .----------------. | | |
| | Nix expression | | | |
| '----------------' | | |
| | | | |
| evaluate | | |
| | | | |
| V | | |
| .------------. | | .------------------. |
| | derivation |----|-instantiate-|->| store derivation | |
| '------------' | | '------------------' |
| | | | |
| | | realise |
fricklerhandwerk marked this conversation as resolved.
Show resolved Hide resolved
| | | | |
| | | V |
| .----------------. | | .--------------. |
| | Nix expression |<-|----read-----|----| store object | |
| '----------------' | | '--------------' |
| | | | |
| evaluate | | |
| | | | |
| V | | |
| .------------. | | |
| | value | | | |
| '------------' | | |
+----------------------+ +------------------------+
```

In more detail, the following sequence diagram shows how the expression is evaluated step by step, and where evaluation is blocked to wait for the build output to appear.

```
.-------. .-------------. .---------.
|Nix CLI| |Nix evaluator| |Nix store|
'-------' '-------------' '---------'
| | |
|evaluate IFD.nix| |
|--------------->| |
| | |
| evaluate `"${readFile drv} world"` |
| | |
| evaluate `readFile drv` |
| | |
| evaluate `drv` as string |
| | |
| |instantiate /nix/store/...-hello.drv|
| |----------------------------------->|
| : |
| : realise /nix/store/...-hello.drv |
| :----------------------------------->|
| : |
| |--------.
| : | |
| (evaluation blocked) | echo hello > $out
| : | |
| |<-------'
| : /nix/store/...-hello |
| |<-----------------------------------|
| | |
| resume `readFile /nix/store/...-hello` |
| | |
| | readFile /nix/store/...-hello |
| |----------------------------------->|
| | |
| | hello |
| |<-----------------------------------|
| | |
| resume `"${"hello"} world"` |
| | |
| resume `"hello world"` |
| | |
| "hello world" | |
|<---------------| |
.-------. .-------------. .---------.
|Nix CLI| |Nix evaluator| |Nix store|
'-------' '-------------' '---------'
```
11 changes: 6 additions & 5 deletions src/libexpr/eval-settings.hh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,12 @@ struct EvalSettings : Config
Setting<bool> enableImportFromDerivation{
this, true, "allow-import-from-derivation",
R"(
By default, Nix allows you to `import` from a derivation, allowing
building at evaluation time. With this option set to false, Nix will
throw an error when evaluating an expression that uses this feature,
allowing users to ensure their evaluation will not require any
builds to take place.
By default, Nix allows [Import from Derivation](@docroot@/language/import-from-derivation.md).

With this option set to `false`, Nix will throw an error when evaluating an expression that uses this feature,
even when the required store object is readily available.
This ensures that evaluation will not require any builds to take place,
regardless of the state of the store.
)"};

Setting<Strings> allowedUris{this, {}, "allowed-uris",
Expand Down