Flakes: Make avoiding dependency diamonds easier somehow.

[TravisWhitaker]

Suppose I have four flakes: A, B, C, and D. Flake A builds an executable, while flakes B, C, and D build libraries. There are two ABI-incompatible versions of D: D1 and D2. Suppose they have the following dependencies:

• A -> B
• A -> C
• B -> D1
• C -> D2

Because the program image of A has two ABI-incompatible versions of D, it could fail to link/segfault/otherwise be broken. This problem tends not to arise with legacy style use of Nixpkgs, since each Nixpkgs rev tends to have one version of each package (with a few exceptions). With flakes, this is no longer true, and I'm not aware of an ergonomic way to deal with this. If I understand flakes correctly, it seems that the author of a flake that depends on library-outputting flakes (like flake A) would need to recursively interrogate their flake input's inputs to avoid making a diamond like this (so that they know what .follows directives to use in their inputs to avoid the problem). There are less catastrophic problems with such deeper flake dependency graphs too: if I have a big closure of dependency flakes that all take nixpkgs as an input, then it's a pain to add all the .follows directives to my inputs, my inputs' inputs, my inputs' inputs' inputs, and so on if I want to e.g. use a single nixpkgs rev everywhere.

I'd propose the following mechanism to help with this problem: provide some way of collecting a flake's input closure, then pin one version of each unique flake in the closure in the flake.lock. This mechanism would fail in the case of conflicting shared dependencies that aren't direct inputs to the top-level flake, like D in the example given above. In this case, one solution would be to just tell the user what .follows directives they need to add to their flake inputs to achieve the one-version-of-each-dependency property. I think a better yet solution would be to enable this behavior in the flake itself, so that evaluation would fail if e.g. the author of A didn't specify a single version of D for B and C to use.

Without something like this, I don't see a way to avoid DLL hell with flakes.

[edolstra]

This isn't really an issue with flakes but with the underlying composition mechanisms (like ELF linking). You can have the same problem within one version of Nixpkgs, e.g. if A, B, C, D1, and D2 are packages in Nixpkgs. Flakes already have overrides and follows to deal with this, e.g. to make flake C use D1. It might be nice to have global overrides though, e.g. to force all nixpkgs flakes to one version.

[TravisWhitaker]

I understand that this is fundamentally a problem with ELF, not flakes. However, if flakes are to be used to build ELF artifacts, then some mechanism to address this problem is required. In Nixpkgs, that mechanism is the convention that for a given rev of Nixpkgs, there is (with a few exceptions) one version of each C/C++ library provided. For flakes, what should that mechanism be? If the answer is "the flake author manually finds the shared indirect dependencies in their dependency closure, then manually pins them all" then it doesn't seem practical to use flakes to distribute C/C++ libraries/binaries. At the very least, it seems the flake author should be able to opt-in to warnings about such shared indirect dependencies so they know what they have to pin to get a stable ABI.

[TravisWhitaker]

Related: #5576

[roberth]

This isn't really an issue with flakes but with the underlying composition mechanisms

It's not just a limitation of ELF. There are also cases where A requires that D1 and D2 data structures are interchangeable.

Requirements like these are part of package management.

I do see potential for mismatches between Flakes' more static metadata and whatever is going on in the derivations and outputs. For instance, a flake could offer two variants of a library. This could make a simple automatic decision at the flake level impossible. So I think it'd be best for API/ABI checks to be done at the derivation level and let flakes manage diamond dependencies where it is simple enough to do so. This way, Flakes can have sensible default behavior in many cases and delegate to the composing flake author when necessary (even if via a build error).

[TravisWhitaker]

So I think it'd be best for API/ABI checks to be done at the derivation level and let flakes manage diamond dependencies where it is simple enough to do so. This way, Flakes can have sensible default behavior in many cases and delegate to the composing flake author when necessary (even if via a build error).

Unfortunately, I think this puts the author/packager of C/C++ code in a weaker position than they would be just using legacy Nixpkgs when it comes to ABI coherence (see https://www.cs.dartmouth.edu/~sergey/cs258/ABI/UlrichDrepper-How-To-Write-Shared-Libraries.pdf), since they now have to do extra manual work to ensure there's ABI coherence among the closure of their dependencies. In the legacy Nixpkgs case, they're saved from this by the convention that Nixpkgs tends to have one version of each C/C++ library. Unless everything is statically linked, I don't see how the author/packager can avoid this manual ABI conflict resolution work

[roberth]

Diamond dependencies are also relevant for NixOS modules and flake-parts modules.

• No great way to close over flake inputs hercules-ci/flake-parts#104