Don't conflate dependencies in setup.sh

[edolstra]

We currently have the confusing situation that the buildInputs attribute gets passed to the builder via the nativeBuildInputs environment variable:

$ nix-shell '<nixpkgs>' -A pan
[nix-shell:~]$ echo $buildInputs 

[nix-shell:~]$ echo $nativeBuildInputs 
/nix/store/aprw7bdaydpxcv6j3wlr9bnr2s061gq6-pkg-config-0.23 /nix/store/75kv0ac0zn6sxypqh4anwwfdmm3bcw35-gtk+-2.24.24 ...

(Pan declares a buildInputs attribute.)

Also, "native" is somewhat ambiguous: native to the build machine, or native to the target machine? However the GNU terminology (build and host) is not really better.

Maybe it would be better to always use buildInputs for the native (build machine) dependencies, and something like targetBuildInputs for the cross-built dependencies. Or not use buildInputs in setup.sh at all and use nativeBuildInputs and targetBuildInputs to prevent confusion.

Ping @viric

[bjornfor]

I agree that "native" is confusing, so +1 for changing that.

But can you explain what is wrong with the GNU terminology? Is there something better? At least the meaning of GNU build/host/target is well documented (and widely used because of autotools), although arguably not intuitive. (But I guess nothing is.)

[viric]

GNU terminology:

• build: the system where the program is being built
• host: the system where the program is to be run
• target: the system the program will handle (as in gdb, binutils, ...)

Current nix terminology in setup.sh env vars:

• nativeBuildInputs: dependencies built for the system where we build $out
  (GNU build)
• buildInputs: dependencies cross-built for the system where $out will run
  (GNU host)
• things like GNU target don't involve dependencies, usually. So, not handled in
  any generic way. We can forget about it, but that may make us careful about
  using the word 'target'

nixpkgs expressions have "buildInputs", because that's the common when not
cross-building. Many expressions can be cross-built fine, by considering its
buildInputs as "GNU host". That's why I thought that converting all buildInputs
as "GNU host dependencies" allows cross-building with no changes to many nixpkgs
autotools-based packages.

So we have these nix var → env var correspondences:

• native:
  buildInputs → nativeBuildInputs
  nativeBuildInputs → nativeBuildInputs
• cross:
  buildInputs → buildInputs
  nativeBuildInputs → nativeBuildInputs

Another option, like what you want, would be to have these correspondences:

• native:
  buildInputs → buildInputs
  nativeBuildInputs → buildInputs
• cross:
  bulidInputs → hostBuildInputs
  nativeBuildInputs → buildInputs

This option, though, makes a weird crossing of names in case of cross building.
Do you prefer this? Do you see any other possibilities?

Regards,
Lluís.

On Thu, Nov 06, 2014 at 02:52:17AM -0800, Eelco Dolstra wrote:

We currently have the confusing situation that the buildInputs attribute gets passed to the builder via the nativeBuildInputs environment variable:

$ nix-shell '<nixpkgs>' -A pan
[nix-shell:~]$ echo $buildInputs 

[nix-shell:~]$ echo $nativeBuildInputs 
/nix/store/aprw7bdaydpxcv6j3wlr9bnr2s061gq6-pkg-config-0.23 /nix/store/75kv0ac0zn6sxypqh4anwwfdmm3bcw35-gtk+-2.24.24 ...

(Pan declares a buildInputs attribute.)

Also, "native" is somewhat ambiguous: native to the build machine, or native to the target machine? However the GNU terminology (build and host) is not really better.

Maybe it would be better to always use buildInputs for the native (build machine) dependencies, and something like targetBuildInputs for the cross-built dependencies. Or not use buildInputs in setup.sh at all and use nativeBuildInputs and targetBuildInputs to prevent confusion.

Ping @viric

---

Reply to this email directly or view it on GitHub:
#4855

[Mathnerd314]

We could drop the 'build' of 'buildInputs', then use GNU prefixes when cross-building:

current	renamed
buildInputs	inputs
nativeBuildInputs	buildInputs
propagatedBuildInputs	propagatedInputs, hostInputs?
propagatedNativeBuildInputs	propagatedBuildInputs

I'm not certain I understand the comment in

nixpkgs/pkgs/stdenv/adapters.nix

Line 102 in 5ad448d

# Cross-linking dynamic libraries, every buildInput should

, since the code applies to all inputs and not just dynamic libraries, but maybe propagateInputs could be hostInputs since it contains all the host dependencies when cross-building. It's also used for things like perl library dependencies, which makes some small amount of sense when called hostInputs.

[domenkozar]

Why not just use cross word instead of target?

[linquize]

@Mathnerd314 refactoring too many files?

[Mathnerd314]

I think this sedscript should be sufficient:

s/buildInput/input/;s/BuildInput/Input/;s/nativeInput/buildInput;s/propagatedInput/hostInput/;s/propagatedNativeInput/propagatedBuildInput/

It catches the other variations, e.g. extraNativeBuildInputs.

[bjornfor]

@viric: Ah, I didn't think of that "GNU target" doesn't involve dependencies. So, when talking about build inputs, we have only two platforms to care about, not three. Good.

@iElectric: I think "cross" is an unfortunate name, because unless you're doing actual cross-compilation, "cross" inputs would be build inputs for the build host, not for the target.

@edolstra: I like both your proposals. But seing as derivations have a .nativeDrv attribute (at least when cross-compiling), I think I'm leaning towards the nativeBuildIputs name. That way we get the most symmetry.

When we rename build inputs, I think it'd be good idea to also rename the .crossDrv derivation attribute to .targetDrv.

[bjornfor]

On the other hand, when doing cross-compilation, the two systems are often called "host" and "target". E.g. Buildroot uses this naming scheme.

[domenkozar]

Relevant: http://lists.science.uu.nl/pipermail/nix-dev/2015-March/016452.html

[edolstra]

To be honest, I think that renaming is going to cause too much work/breakage at this point. For the overrideDerivation issue, it would be enough to move the concatenating of buildInputs to nativeBuildInputs into the generic builder script.

[bjornfor]

Oh, don't say that! :-) What kind of breakage? Internal to nixpkgs or for users with out-of-tree expressions?

I'm very much in favor of a rename, some time in the future.

[akaWolf]

Any progress?

[lucabrunox]

Alternative approach. Because packages like flex, pkgconfig, ecc. are usually (or always) used as nativeBuildInputs, what about setting for them a nix-support/native-build-input ?

We get rid of native vs cross build inputs, and instead each package declare itself as native vs cross build input, or in general its role in the build process.

Otherwise it's a maintainance burden to understand which package goes native vs cross, or also renaming this kind of variables. We'll only have one buildInputs.

In general tools are native, libraries are cross. The multi output branch should even help with this distinction, of course with the possibility to override the role.

It's not much useful to keep using native vs cross, if one (like me) is not going or forget to use it. So just let's make this automatic.

[edolstra]

@lethalman 👍

[nbp]

@lethalman 👍 I think this makes sense, but you probably want to consider the lack of the file as a proof of a native packages versus a cross-compiled package,

Thus having nix-support/host-system which contains name of the system on which these binaries/libraries can be executed safely, might makes more sense. Note, that having the name of the host-system is better if we want to import these packages in the nix store of another computer.

Thus the rule would be that if the file does not exists, we assume that this is the same system, and if it does, then we check against the system on which we are building.

[dezgeg]

I don't see how this could be solved via nix-support/* files; it definitely needs to be done on the nix expression level. Consider that when cross compiling, nativeBuildInputs = [foo] and buildInputs = [foo] cause entirely different foo derivations to be added as dependencies.

[bjornfor]

A data point, how the Buildroot embedded build system solve this:

Buildroot has two types of dependencies, one for normal dependencies (for the target architecture) and one for the (build) host. Both go into the same dependency list:

# libfoo.mk
LIBFOO_DEPENDENCIES = host-libaaa libbbb
...

The package "expression" for libaaa (a Makefile snippet) decides what type of packages are created:

# libaaa.mk
... metadata and dependency info
$(eval $(autotools-package))  # create a normal (target) package
$(eval $(host-autotools-package))  # also create a build host package

I like the explicitness of this approach. I'd like to see this in nixpkgs:

  buildInputs = [ libfoo.hostDrv libbbb ];

[lucabrunox]

@bjornfor that doesn't change much the current state, you put everything in buildInputs instead of having two sets, but that's it.

@dezgeg it does, because bison will always be used as host dependency, and glib always as target dependency.

[bjornfor]

But some packages can be used both as target and host inputs. So I think there is value in having explicit distinction between host/target derivations (i.e. libfoo vs libfoo.hostDrv).

[lucabrunox]

@bjornfor that's the exception, and we already have that... it's called getNativeDrv drv.

[lucabrunox]

To be clear, pkgs.bison will be already the nativeDrv @dezgeg , pkgs.bison = getNativeDrv (callPackage ...). So we probably don't even need the nix-support thing, but it would be useful to have the information in nix-support still.

[twhitehead]

+1 regarding fixing the overrideDerivation behaviour. Just finished running into this adding libibverb as a dependency to openmpi to get infiniband support.

openmpi.overrideDerivation (oldattrs: {
  buildInputs = oldattrs.buildInputs ++ [ libibverbs ];
} )

gave a derivation with buildInputs set to the extra libibverbs

Derive(
  ...
, [ ("buildInputs", "/home/nixbld/store/3ygnx9458la0m22m7250j2qwkrsgzbxp-libibverbs-1.1.8")
    ...
  , ("nativeBuildInputs", "/home/nixbld/store/ww3a97ggvdck01kjmlffj3wb27gg7xqr-gfortran-wrapper-4.9.3")
  ...
  ]
)

while directly modifying buildInputs in the openmpi/default.nix file gave a derivation with nativeBuildInputs set with the extra libibverbs.

Derive(
  ...
, [ ("buildInputs", "")
    ...
  , ("nativeBuildInputs", "/home/nixbld/store/ww3a97ggvdck01kjmlffj3wb27gg7xqr-gfortran-wrapper-4.9.3 /home/nixbld/store/3ygnx9458la0m22m7250j2qwkrsgzbxp-libibverbs-1.1.8")
   ...
  ]
)

[dezgeg]

AFAICT solving #10721 will solve the overrideDerivation problem as well.

[domenkozar]

No consensus how to move forward here, bumping milestone.

[DamienCassou]

Any news?

[domenkozar]

This won't make it to 16.09

[aneeshusa]

#18660 adds a new overrideAttrs function, which is essentially a fixed version of overrideDerivation.

[Ericson2314]

In #26805 I do add 4 more sorts of dependencies giving us:

Current name	Ideal name	Host is	Target is
depsBuildBuild	depsBuildBuild	build platform	build platform
nativeBuildInputs	depsBuildHost	build platform	host platform
__depsBuildTarget	__depsBuildTarget	build platform	target platform
__depsHostHost	__depsHostHost	host platform	host platform
buildInputs	depsHostTarget	host platform	target platform
__depsTargetTarget	__depsTargetTarget	target platform	target platform

Packages do not care what the build platform of the dep is, because they merely need it to exist. In practice however, the build platform of the dep is the build platform of the consumer.

[ip1981]

The build-host-target triad is quite clear, but target is not applicable to the most of software, it's only relevant to compilers:

• build is where we build/compile software.
• host is where this software will run.
• target is where software built by this software will run :)

E. g. we can build GCC on linux (build), which will run on Windows (host), but compile binaries for solaris (target).

[Ericson2314]

@ip1981 That's all true. Precisely because "target" is not applicable to most software, yet that's not clear from the name alone, I prefixed with underscores the "bad" dependencies to discourage accidental use.

[Ericson2314]

N.B. Editted my table now that #26805 now introduces even more dependencies.

[dezgeg]

FWIW, the original problem (buildInputs attribute gets passed to the builder via the nativeBuildInputs environment variable) by @edolstra has been solved by #23374 and #10721 for quite a while now. Should it be time to close this and split off the other cross naming issues into separate tickets?

[Ericson2314]

@dezgeg Sure, that makes sense to me.

[Ericson2314]

Remaining bike-shed is in #28327