deterministic-uname: init #210102

Artturin · 2023-01-10T20:57:20Z

for reproducibility

based on #64258

Closes #140175 and probably more

Description of changes

Things done

pkgs/build-support/fake-uname/fake-uname.sh

Artturin · 2023-01-10T21:16:14Z

pkgs/build-support/fake-uname/fake-uname.sh

+MACHINE_VAL=@processor@
+PROCESSOR_VAL=unknown
+HARDWARE_PLATFORM_VAL=unknown
+OPERATING_SYSTEM_VAL=GNU/Linux


we should add uname -o (-o, --operating-system print the operating system) to

nixpkgs/lib/systems/default.nix

Lines 88 to 115 in 2377605

# Output from uname

uname = {

# uname -s

system = {

linux = "Linux";

windows = "Windows";

darwin = "Darwin";

netbsd = "NetBSD";

freebsd = "FreeBSD";

openbsd = "OpenBSD";

wasi = "Wasi";

redox = "Redox";

genode = "Genode";

}.${final.parsed.kernel.name} or null;

# uname -m

processor =

if final.isPower64

then "ppc64${lib.optionalString final.isLittleEndian "le"}"

else if final.isPower

then "ppc${lib.optionalString final.isLittleEndian "le"}"

else if final.isMips64

then "mips64" # endianness is *not* included on mips64

else final.parsed.cpu.name;

# uname -r

release = null;

};

@amjoseph-nixpkgs

~~IMHO there is no reason for a nixlang expression to need the value of uname -o at eval-time, is there?~~

Maybe put the table with those values in this package so people don't use it from random expressions without a good reason. If a good reason appears we can always move it out of fake-uname into lib/systems.

Edit: sorry, I had this confused with uname -a.

Let me look into this.

Mindavi

I approve of this as it's a lot less controversial that outright replacing uname everywhere. It might be a bit more annoying to apply to the right packages (less scalable), but it's a good first step towards solving impurity issues with uname. If this is applied in too many places we can always reconsider the replacement option.

Mindavi · 2023-01-10T21:18:05Z

pkgs/development/libraries/libgtop/default.nix

@@ -26,6 +27,7 @@ stdenv.mkDerivation rec {
    perl
    gettext
    gobject-introspection
+    fake-uname


How is it ensured that this uname takes precedence over other uname implementations?

nixpkgs/pkgs/stdenv/generic/common-path.nix

Line 2 in 6abb95c

pkgs.coreutils

nixpkgs/pkgs/stdenv/linux/default.nix

Lines 492 to 493 in 6abb95c

initialPath =

((import ../generic/common-path.nix) {pkgs = prevStage;});

nixpkgs/pkgs/stdenv/generic/default.nix

Lines 133 to 134 in 6abb95c

inherit initialPath shell

defaultNativeBuildInputs defaultBuildInputs;

nixpkgs/pkgs/stdenv/generic/setup.sh

Lines 274 to 287 in 6abb95c

# Set up the initial path.

PATH=

HOST_PATH=

for i in $initialPath; do

if [ "$i" = / ]; then i=; fi

addToSearchPath PATH "$i/bin"

# For backward compatibility, we add initial path to HOST_PATH so

# it can be used in auto patch-shebangs. Unfortunately this will

# not work with cross compilation.

if [ -z "${strictDeps-}" ]; then

addToSearchPath HOST_PATH "$i/bin"

fi

done

somehow PATH contains the nativeBuildInputs first and then common-path

declare -x PATH="/nix/store/k2vd7w2ixbqbgd9v88am8780111bng7j-pkg-config-wrapper-0.29.2/bin:/nix/store/lw5mv8qzyalks7w80a6615na48i8lny1-gtk-doc-1.33.2/bin:/nix/store/n48i43wkly6kfhxz4ypcxbsv2pr7ggzh-perl-5.36.0/bin:/nix/store/zms41v0fgzfq4z629kn5jrlcv7hf9w7v-gettext-0.21/bin:/nix/store/3qgzjk0i8g7gif5jly94nbph0jwx4d0s-gobject-introspection-wrapped-1.74.0-dev/bin:/nix/store/02ggkplpkvdjkr20bylnidhn7cyh66m4-glib-2.74.3-dev/bin:/nix/store/k886wb6j8m1jwxf5pf1d5mdmlyblyd1y-glib-2.74.3-bin/bin:/nix/store/g2nfbpind37b3v3y6zbszmcgfmrj5v7x-patchelf-0.15.0/bin:/nix/store/wn31i3dzwahz6ccws8bs1nwyqrpgsvj7-gcc-wrapper-11.3.0/bin:/nix/store/sxdx80lmk4zkhb51f4x5dgqvxgmx55wl-gcc-11.3.0/bin:/nix/store/s7ip867mrpnnjlppbnxlcsq10gv13x2x-glibc-2.35-224-bin/bin:/nix/store/h8gvq6r4hgpa71h44dmg9qfx03mj81sv-coreutils-9.1/bin:/nix/store/62fb427ncxaaksa2k59rhbilfg68v20x-binutils-wrapper-2.39/bin:/nix/store/frh9l9nrdysasdi2gs7i241s241ngjw2-binutils-2.39/bin:/nix/store/w73bxii8xlsm9459628glawfcjlci75d-gobject-introspection-1.74.0-dev/bin:/nix/store/h8gvq6r4hgpa71h44dmg9qfx03mj81sv-coreutils-9.1/bin:/nix/store/zml88vnkpm8if114qkbbqd1q7n3ypqqy-findutils-4.9.0/bin:/nix/store/49y3r0gr9m6k20d91kl6dgp4b9a6m72v-diffutils-3.8/bin:/nix/store/5dv5cq1lwvsijr9p16p2kp79g1dbajk3-gnused-4.8/bin:/nix/store/bcvccw6y9bfil6xrl5j7psza7hnd16ry-gnugrep-3.7/bin:/nix/store/l1fp0hyca54xbb85vfhppd16bskzx8dg-gawk-5.1.1/bin:/nix/store/89zbjdkb48ma61k76l2mzn3s0ra0wn2c-gnutar-1.34/bin:/nix/store/qs8qb1swpivkfq7i9yd52n0mw6z4ij81-gzip-1.12/bin:/nix/store/wwkyfg8b34xy16zzc9p6rkh59p4q37qx-bzip2-1.0.8-bin/bin:/nix/store/i0x4pzj96qwvkrm94317l6jbi53a2rdj-gnumake-4.4/bin:/nix/store/4xw8n979xpivdc46a9ndcvyhwgif00hz-bash-5.1-p16/bin:/nix/store/793iwfbjvg7wgpqq7r83a1qjl1yg02sf-patch-2.7.6/bin:/nix/store/a3mwv26f99ycsv9w6hrx0jjjjywvcb1n-xz-5.2.9-bin/bin:/nix/store/camlh5laf1wsklghk0vcaw7gvx4rpzd1-file-5.43/bin"

if someone does this then the fake-uname won't be used but if coreutils is after fake-uname then the fake is used

diff --git a/pkgs/development/libraries/libgtop/default.nix b/pkgs/development/libraries/libgtop/default.nix index 59db7929240..8a256e5f811 100644 --- a/pkgs/development/libraries/libgtop/default.nix +++ b/pkgs/development/libraries/libgtop/default.nix @@ -8,6 +8,7 @@ , gnome , gtk-doc , fake-uname +, coreutils }: stdenv.mkDerivation rec { @@ -27,6 +28,7 @@ stdenv.mkDerivation rec { perl gettext gobject-introspection + coreutils fake-uname ];

Or possibly when one of the preceding dependencies propagates it?

pkgs/build-support/fake-uname/default.nix

jtojnar · 2023-01-10T21:40:16Z

pkgs/development/libraries/libgtop/default.nix

@@ -26,6 +27,7 @@ stdenv.mkDerivation rec {
    perl
    gettext
    gobject-introspection
+    fake-uname


There needs to be a comment that this is to ensure reproducibility. Ideally with some details so that the necessity can be judged in the future.

Maybe fake-uname should be called deterministic-uname or uname-deterministic.

chose deterministic-uname and added details as a comment

pkgs/build-support/fake-uname/default.nix

pkgs/build-support/fake-uname/fake-uname.sh

ghost

Apparently github no longer lets you leave resolvable comments without a "header" comment? A glitch in the matrix.

for reproducibility deterministic-uname: dont hardcode OPERATING_SYSTEM_VAL to GNU/Linux

this will make the build reproducible libgtop_server2 contained Linux 5.15.68 x86_64 This libgtop was compiled on %s %s %s

Artturin · 2023-01-12T16:30:38Z

pkgs/build-support/deterministic-uname/default.nix

+  # uname -o
+  # maybe add to lib/systems/default.nix uname attrset
+  # https://github.com/coreutils/coreutils/blob/7fc84d1c0f6b35231b0b4577b70aaa26bf548a7c/src/uname.c#L373-L374
+  # https://stackoverflow.com/questions/61711186/where-does-host-operating-system-in-uname-c-comes-from
+  # https://github.com/coreutils/gnulib/blob/master/m4/host-os.m4
+  operatingSystem =
+    if stdenv.buildPlatform.isLinux
+    then "GNU/Linux"
+    else if stdenv.buildPlatform.isDarwin
+    then "Darwin" # darwin isn't in host-os.m4 so where does this come from?
+    else "unknown";


Artturin · 2023-01-13T18:26:12Z

sidenote

coreutils uname has basic supports for env vars but only on darwin since coreutils/coreutils@e3d1ff0

Artturin · 2023-01-13T19:26:43Z

#210569

ajs124 · 2023-01-14T14:08:46Z

This seems to have broken the tarball job:

Nixpkgs is not allowed to use <nixpkgs> to refer to itself.
The offending files: /nix/store/79pm0hmza6jwi2swa62kzlh549ivy9jh-source/pkgs/build-support/deterministic-uname/default.nix

Artturin · 2023-01-14T14:22:30Z

pkgs/build-support/deterministic-uname/default.nix

+  modDirVersion = if modDirVersion != "" then modDirVersion else "unknown";
+
+  meta = with lib; {
+    description = "Print certain system information (hardcoded with <nixpkgs/lib/system> values)";


@ajs124 it's catching the <> here, it should be changed to with lib/system values

misuzu · 2023-01-26T20:04:43Z

pkgs/build-support/deterministic-uname/deterministic-uname.sh

+    echo -n "$KERNEL_NAME_VAL $NODENAME_VAL $KERNEL_RELEASE_VAL $KERNEL_VERSION_VAL $MACHINE_VAL"
+    # in help:  except omit -p and -i if unknown.
+    #echo -n "$PROCESSOR_VAL $HARDWARE_PLATFORM_VAL\n"
+    echo -n "$OPERATING_SYSTEM_VAL"


Whitespace is missing:

$ nix shell nixpkgs#deterministic-uname -c uname -a Linux nixpkgs unknown #1-NixOS Tue Jan 1 00:00:00 UTC 1980 x86_64GNU/Linux

thanks fix in #213768

misuzu · 2023-01-26T20:16:52Z

Would be awesome if this could be used in low-level packages (perl for example, adding deterministic-uname to nativeBuildInputs triggers infinite recursion) to build NixOS for armv7l-linux on aarch64 machines without patching the kernel.

Artturin · 2023-01-31T13:07:23Z

Would be awesome if this could be used in low-level packages (perl for example, adding deterministic-uname to nativeBuildInputs triggers infinite recursion) to build NixOS for armv7l-linux on aarch64 machines without patching the kernel.

for that we need to get rid of the coreutils and getopt dependencies or do something with bootstrapTools

Artturin added the 6.topic: reproducible builds label Jan 10, 2023

Artturin requested a review from primeos January 10, 2023 20:57

Artturin marked this pull request as draft January 10, 2023 20:59

Artturin force-pushed the fakeunameinit branch from 3994f82 to 8c7a38c Compare January 10, 2023 21:09

Artturin mentioned this pull request Jan 10, 2023

libgtop is not reproducible #140175

Closed

Artturin requested a review from Mindavi January 10, 2023 21:12

Mindavi reviewed Jan 10, 2023

View reviewed changes

pkgs/build-support/fake-uname/fake-uname.sh Outdated Show resolved Hide resolved

Artturin force-pushed the fakeunameinit branch from 8c7a38c to b06592c Compare January 10, 2023 21:14

Artturin commented Jan 10, 2023

View reviewed changes

Artturin force-pushed the fakeunameinit branch from b06592c to 6897cba Compare January 10, 2023 21:19

Mindavi reviewed Jan 10, 2023

View reviewed changes

ofborg bot added the 8.has: package (new) label Jan 10, 2023

ofborg bot requested review from dasj19, amaxine, hedning, jtojnar and bobby285271 January 10, 2023 21:31

ofborg bot added 10.rebuild-darwin: 11-100 10.rebuild-linux: 101-500 labels Jan 10, 2023

jtojnar reviewed Jan 10, 2023

View reviewed changes

Artturin force-pushed the fakeunameinit branch 3 times, most recently from 3afdebf to 20cde9e Compare January 10, 2023 21:41