Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable debug symbols for all packages #18530

Open
stapelberg opened this issue Sep 12, 2016 · 63 comments
Open

Enable debug symbols for all packages #18530

stapelberg opened this issue Sep 12, 2016 · 63 comments

Comments

@stapelberg
Copy link
Contributor

PR #15539 mentions this, but I think it’s worthwhile to have a separate GitHub issue.

Currently, a large number of libraries are lacking debug symbols even though environment.enableDebugInfo = true; is specified in my /etc/nixos/configuration.nix. The following list is taken from a gdb session on NetworkManager:

(gdb) info sharedlibrary 
From                To                  Syms Read   Shared Object Library
0x00007ffff7ddaab0  0x00007ffff7df5990  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/ld-linux-x86-64.so.2
0x00007ffff7bd6630  0x00007ffff7bd7d01  Yes (*)     /nix/store/33z8lb3yzh18z10aqszc07imwlfkyhgb-util-linux-2.28.1/lib/libuuid.so.1
0x00007ffff78c7c80  0x00007ffff7993a44  Yes (*)     /nix/store/3yvgy2mnsm2cfvbjkdcaxlzg1xwh80fj-gnutls-3.4.14/lib/libgnutls.so.30
0x00007ffff764b830  0x00007ffff767a1cf  Yes (*)     /nix/store/yzb1maz797c80606aqj78hafbc4lidcr-p11-kit-0.23.2/lib/libp11-kit.so.0
0x00007ffff7408fd0  0x00007ffff740d6e6  Yes (*)     /nix/store/4b33d5w92qxaw464kgyfs6ncskwmp056-libidn-1.33/lib/libidn.so.11
0x00007ffff71f5d50  0x00007ffff7200b6b  Yes (*)     /nix/store/13lpz809nmpzl12hmn09xnd6yk2jvy7s-libtasn1-4.8/lib/libtasn1.so.6
0x00007ffff6fc4ee0  0x00007ffff6fe15ef  Yes (*)     /nix/store/5rqh0fg8004z9gkwvywbpg75ydf3w9gy-nettle-3.1.1/lib/libnettle.so.6
0x00007ffff6d8f270  0x00007ffff6d9c1e8  Yes (*)     /nix/store/5rqh0fg8004z9gkwvywbpg75ydf3w9gy-nettle-3.1.1/lib/libhogweed.so.4
0x00007ffff6b04280  0x00007ffff6b6ee68  Yes (*)     /nix/store/gb81bp3xbwrhkqd8cw7cvwxl0qlzah4x-gmp-6.1.1/lib/libgmp.so.10
0x00007ffff68edb90  0x00007ffff68f0c62  Yes (*)     /nix/store/z9nhrhcgiqcag9frhdz8lwriypvl8kc5-libgudev-230/lib/libgudev-1.0.so.0
0x00007ffff7fd5c60  0x00007ffff7fe6886  Yes (*)     /nix/store/lamh95q185ymigm217s64x08qcmnfhvm-systemd-231/lib/libudev.so.1
0x00007ffff66d13c0  0x00007ffff66deb35  Yes (*)     /nix/store/vw0wcq2zz1w3l86ly5pzw2dphfpychym-libnl-3.2.28/lib/libnl-3.so.200
0x00007ffff7f59760  0x00007ffff7fae2aa  Yes (*)     /nix/store/lamh95q185ymigm217s64x08qcmnfhvm-systemd-231/lib/libsystemd.so.0
0x00007ffff64c4fe0  0x00007ffff64c6784  Yes (*)     /nix/store/z4mg6wiy5y26p0djr95b4b6cnpf3m7qb-libndp-1.6/lib/libndp.so.0
0x00007ffff622a2e0  0x00007ffff6270aef  Yes (*)     /nix/store/3dg2ld3q204xsi2ml12073nw9bng7xf7-libsoup-2.54.1/lib/libsoup-2.4.so.1
0x00007ffff5eb07a0  0x00007ffff5f93160  Yes (*)     /nix/store/9didfvj6wba723zcnls1c5iwp9jv3l3l-libxml2-2.9.4/lib/libxml2.so.2
0x00007ffff5b83660  0x00007ffff5beffba  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libm.so.6
0x00007ffff58b2440  0x00007ffff5955100  Yes (*)     /nix/store/1bx29pqdzwc722an88nzxxvn24al2nyb-sqlite-3.14.1/lib/libsqlite3.so.0
0x00007ffff5555490  0x00007ffff562892b  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libgio-2.0.so.0
0x00007ffff531d2b0  0x00007ffff531e1b5  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libgmodule-2.0.so.0
0x00007ffff51081a0  0x00007ffff51143d9  Yes (*)     /nix/store/ahjpbwi957n7pqp7i7ys6s2vxmk4dr06-zlib-1.2.8/lib/libz.so.1
0x00007ffff4ef2810  0x00007ffff4efe93b  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libresolv.so.2
0x00007ffff4ca79e0  0x00007ffff4cd83a6  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libgobject-2.0.so.0
0x00007ffff4a95aa0  0x00007ffff4a9a35c  Yes (*)     /nix/store/3zzw0gc6078nsp829x2va9690kca237j-libffi-3.2.1/lib/../lib64/libffi.so.6
0x00007ffff479ef80  0x00007ffff4815bb9  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libglib-2.0.so.0
0x00007ffff4516730  0x00007ffff4565996  Yes (*)     /nix/store/10jk5h18ahh2xaad74ng3mgx0lfjmfj6-pcre-8.38/lib/libpcre.so.1
0x00007ffff42fd810  0x00007ffff430a531  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libpthread.so.0
0x00007ffff40f4d70  0x00007ffff40f590e  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libdl.so.2
0x00007ffff3d75930  0x00007ffff3e9e1a3  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libc.so.6
0x00007ffff3b526d0  0x00007ffff3b53fd6  Yes (*)     /nix/store/4am908pjq2j4bzqcy051mx62ab0rd1xa-libcap-2.25-lib/lib/libcap.so.2
0x00007ffff394af70  0x00007ffff394dd1f  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/librt.so.1
0x00007ffff3725e30  0x00007ffff373bff2  Yes (*)     /nix/store/3qq5qv97jdqg0ywq14spj7kmif39fhnw-xz-5.2.2/lib/liblzma.so.5
0x00007ffff35132a0  0x00007ffff351f5bb  Yes (*)     /nix/store/bppnjqhm8jjbwc8215i9qvg4ws39avz1-lz4-131/lib/liblz4.so.1
0x00007ffff320ce00  0x00007ffff32cce48  Yes (*)     /nix/store/i7z7i541bfa6gxw1g71wzf82gj5ms8s7-libgcrypt-1.7.3/lib/libgcrypt.so.20
0x00007ffff2ff0a30  0x00007ffff2ffa099  Yes (*)     /nix/store/4ixpz8hqaqv4h4hibm0m8ljpbv38wd3n-libgpg-error-1.24/lib/libgpg-error.so.0
0x00007fffea9c30d0  0x00007fffea9c8d91  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libnss_files.so.2
0x00007fffea7a6800  0x00007fffea7b74c6  Yes         /tmp/nm/nix/store/g05c3qil03b5lg02nqlz5viv2ad3bsam-network-manager-1.2.2/lib/NetworkManager/libnm-device-plugin-wifi.so
(*): Shared library is missing debugging information.

In the above list, only glibc seems to come with debug symbols, no other library does.

https://nixos.org/nixos/manual/options.html#opt-environment.enableDebugInfo explains how to override individual packages, but that’s tedious and requires recompilation of all reverse-dependencies, which is really really inconvenient (my machine is compiling spidermonkey since half an hour… :-/).

Debian recently started building debug symbol paackages by default and Fedora seems to have debuginfo RPMs built by default as well. I think having debug symbols on by default is a feature that users of modern Linux distributions have come to expect :).

Could we enable debug symbols for all packages by default please?

cc @kevincox

@edolstra
Copy link
Member

I don't know if we should enable it for all packages. We should probably have some guesstimate of the additional storage costs. However, it's definitely a good idea to enable it for some major libraries.

@stapelberg
Copy link
Contributor Author

I’m assuming you’re referring to the build output stored on build servers, since the debug info on the disks of end users is configurable using enableDebugInfo :).

For comparison, here’s a list of the sizes (in bytes) of all 1726 Debian packages with a “-dbg” suffix currently in Debian testing (amd64): debian-debug-sizes.txt
The sum of all of these packages is about 10 GiB.

Here’s a list of the sizes (in percent, e.g. 100 is 100%) relative to the corresponding package: relative.txt
The average size of a debug package is 4557% of the base package.

Note that these should be treated as rough estimates: the base package matching was done textually (as opposed to understanding the source-package relationship).

Based on these values from Debian, can you do a guesstimate as to how much additional space would be required? AFAICT (without any knowledge of the NixOS build infrastructure), the additional demand seems to range in the tens of gigabytes region, which should be totally doable…?

@kevincox
Copy link
Contributor

I would love to see this happen. As for my other PR life got in the way and there was debate that never got resolved. The basics work but you can see the checkboxes to see what features I would like added (although they can likely be added as a later patch). I should have time to pick it up around now, hopefully banging it into a shape that is generally agreed to be acceptable.

As for the package sizes this is obviously a code tradeoff and it will depend on what hydra/S3 can have available. It would definitely be awesome to have most libraries built with debug symbols.

@copumpkin
Copy link
Member

Would we put them in a separate output?

@kevincox
Copy link
Contributor

On Sep 12, 2016 23:08, "Daniel Peebles" notifications@github.com wrote:

Would we put them in a separate output?

I'm pretty sure this is a requirement.

@vcunat
Copy link
Member

vcunat commented Sep 13, 2016

Would we put them in a separate output?

It should all be just a matter of adding separateDebugInfo = true; into the particular packages.

Note: the debian list you provided has 1726 lines. One Hydra evaluation of ours contains way more builds (~35k). There are some duplicates in there, and many would only have negligible debug info, but there still might be a significant difference. I can't see how to find easily, except to build one generation in that way.

@stapelberg
Copy link
Contributor Author

I’m aware of that. Debian has >20k source packages, so we’re talking about a roughly 10% sample size here at best — that should be representative enough for a guesstimate.

I agree that the easiest way to get accurate numbers would be to just build a generation with debug symbols enabled, though :).

@vcunat
Copy link
Member

vcunat commented Sep 13, 2016

What's their total size of non-debug stuff on this list? (EDIT: ratio of sums or averages seems much more reliable than average of ratios.)

@stapelberg
Copy link
Contributor Author

It’s 8.19G of debug packages and 0.97G of corresponding non-debug packages.

@vcunat
Copy link
Member

vcunat commented Sep 13, 2016

Well, 9-fold increase of required space would be a disaster. Hopefully they don't compress their debug files; we do it.

@stapelberg
Copy link
Contributor Author

On the package level, compression is not accounted for in my numbers: the sizes are for the unpacked, uncompressed package contents.

On the debug file level, I can’t tell whether they are compressed or not…?

$ file /usr/lib/debug/.build-id/ee/13cb8a18d72bea00fd7f7ff22544d3024eb5c3.debug
/usr/lib/debug/.build-id/ee/13cb8a18d72bea00fd7f7ff22544d3024eb5c3.debug: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter *empty*, for GNU/Linux 2.6.32, BuildID[sha1]=ee13cb8a18d72bea00fd7f7ff22544d3024eb5c3, not stripped

@vcunat
Copy link
Member

vcunat commented Sep 14, 2016

Hmm, I see the compression doesn't make that great difference (~30 %) and even after it the debug files are expected to be 2–4x size of the corresponding binaries (for complex C++ at least). https://gcc.gnu.org/wiki/DebugFission But we store more than just ELFs.

We've got glibc example where the increase is ~40 %:

22M     /nix/store/6fix3zqpnahyml8zp2sxi2rwan55rgb8-glibc-2.24
2.7M    /nix/store/g2f08f2gb4dpc2l0g57jy6yp26bfc4hj-glibc-2.24-bin
14M     /nix/store/2gfgppppk96xi91xv90xlz85z73m8prj-glibc-2.24-debug
3.2M    /nix/store/8sl4jfs3nq0pkq4gg655s3axrxdx7z29-glibc-2.24-dev
8.2M    /nix/store/m2mvna9pk9hv1qgxwi124avzdq1i5q96-glibc-2.24-static
50M     total

Perhaps if we restricted the system-wide enabling to x86_64-linux for now, the total increase might be as small as 10 %.

@vcunat vcunat added this to the 17.03 milestone Sep 16, 2016
@vandenoever
Copy link
Contributor

Having debug symbols enabled should be the default in my opinion, especially in a partially self-compiled distributions like NixOS. The threshold to submitting bugs should be low. Great to see it as a milestone for 17.03.

@globin globin modified the milestones: 17.09, 17.03 Mar 13, 2017
@0xABAB
Copy link
Contributor

0xABAB commented Jul 7, 2017

Storage costs also depend on the device being used to store them on. On some systems, I would much rather store the debugging symbols on cheap hard disk space as opposed to an NVME or SSD disk, while keeping the rest of the Nix store on fast storage.

For me it would be the difference between enabling debug symbols for everything and not.

@kevincox
Copy link
Contributor

kevincox commented Jul 9, 2017

@0xABAB I suspect that you wouldn't install debugging symbols on most machines, they would be kept on your binary cache until you need to symbolize a stack trace. This would likely be done on developer machines or dedicated servers.

@0xABAB
Copy link
Contributor

0xABAB commented Jul 9, 2017

@kevincox In that case the specific feature being discussed here is not clear.

As a developer oriented user, I would like to:

  1. run a program with a segfault
  2. get a dialog that it has crashed
  3. have an option in configuration.nix to say to automatically download debugging symbols on demand
  4. see a beautiful back trace which in KDE's user interface has three stars or contains no question marks/hidden symbols in simple English.
  5. purge these downloaded debugging symbols automatically after two weeks if free disk space is below 30%.

I can imagine an operator of a binary cache to want low operational costs, but it seems that with a properly designed Nix the costs are only dependent on the costs of S3, not of the server's attached storage.

I don't see why any user would ever be required to build packages. @volth apparently has reasons to believe there are still cases like that, but I don't see them.

@domenkozar
Copy link
Member

I think the next step is rebuilding nixos-small with debug on and compressing all debug outputs to xz to see how much each package size increases on average.

@edolstra
Copy link
Member

.debug files are already compressed using --compress-debug-sections=zlib. (Whether this is useful is questionable, given that NARs in the binary cache are already compressed using xz, so probably we just end up with a worse compression ratio...)

Note that since a few months, cache.nixos.org provides a index of .debug files (indexed by their build ID). For example, to get the debug symbols for libc.so.6:

$ readelf -a /nix/store/kjwbqnh13dxh6w4pk2gb3ddmhpiaihqg-glibc-2.25/lib/libc.so.6 | grep 'Build ID'
    Build ID: 443f38a7db6510d81a7afe03b1a42de8ef6c6ebf

$ curl https://cache.nixos.org/debuginfo/443f38a7db6510d81a7afe03b1a42de8ef6c6ebf
{"archive":"../nar/0xx62iin8dpql0fn70ad7fy8d2w4vzc10d7nbl84al8k723np3p4.nar.xz","member":"lib/debug/.build-id/44/3f38a7db6510d81a7afe03b1a42de8ef6c6ebf.debug"}

So https://cache.nixos.org/nar/0xx62iin8dpql0fn70ad7fy8d2w4vzc10d7nbl84al8k723np3p4.nar.xz contains the desired debug symbols.

I also wrote a little FUSE file system that fetches debug symbols automatically from the binary cache. So you can mount the file system on (say) /run/debug, point the NIX_DEBUG_INFO_DIRS environment variable to it, and gdb and eu-stack will be able to obtain debug info for any pacakge in the binary cache that has debug info enabled.

@domenkozar
Copy link
Member

@edolstra if I understand correctly one needs to recompile nixpkgs with separateDebugInfo = true to get this working right?

@edolstra
Copy link
Member

@domenkozar Yes.

@fpletz fpletz removed this from the 17.09 milestone Mar 4, 2018
@Ericson2314
Copy link
Member

I think we're making a mistake conflating nixpkgs and cache.nixos.org policy here. Even if we never stored debug outputs, but still built them, it would be useful to CI that the building still works. I'm OKish doing a mass rebuild for myself to get those outputs, but only if I'm sure that mass rebuild won't be in vain because it isn't tesed and bitrots.

@FRidh FRidh modified the milestones: 20.09, 21.03 Dec 20, 2020
@poelzi
Copy link
Member

poelzi commented Jan 12, 2021

Debugging on NixOS is one of the most frustrating experiences ever and drive me to the point where I want to ditch NixOS again.
Debug symbols for everything need to be install-able and working for all packages and especially libs without days of trial and error.
Status quo is just broken and useless, better debug your stuff on a debian/arch/whatever VM then try to get this stuff on nix built...

@Gaelan
Copy link
Contributor

Gaelan commented Mar 7, 2021

Would it be possible, as a compromise/interim solution, to have Hydra build packages with debug information enabled (as an extra output), but then ditch the extra output as soon as it's built? That would eliminate the disk space requirements on Hydra, but since things would be in the "with debug info" store paths, it would be possible to obtain debug info for a Hydra-build package by "simply" rebuilding the package locally. We could even have third parties/community members distribute debug info for some packages if there is interest.

@stale
Copy link

stale bot commented Sep 6, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Sep 6, 2021
@TredwellGit TredwellGit removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Sep 13, 2021
@oxalica
Copy link
Contributor

oxalica commented Nov 19, 2021

I had a hard time when debugging firefox issues recently due to the lack of debug symbols.

I also ran into the issue of separateDebugInfo not recognizing artifacts from LLVM toolchain when trying to enable debug symbols for firefox. See #146275

@fzakaria
Copy link
Contributor

fzakaria commented Jan 4, 2022

Want to chime in here.

I went through the process of building glibc with debug symbols to triage a linker error.
The documentation in https://nixos.wiki/wiki/Debug_Symbols was amazing; as an aside it should definitely work it's way into the official documentation.

I definitely agree that conflating building by default all debug symbols and whether or not to store them are two separate things.
Having potential automatic debug symbols for NixOS / Nix would be such a game changer, and would also highlight the power of the fact that it's a source based build system.

Perhaps even the $$ discussion can be revisited. Hopefully after the years since this ticket has been open, the monetary outlook and funding of NixOS is different and this can be reviewed.

Doing this for only x68-64 to start is also extremely sensible.

Two improvements I could see as well:

  1. enableDebugging that also perhaps turns off "separateDebugInfo". Sometimes it's easier just to have it in the binary
  2. enableDebugging that also applies itself transitively to all build dependencies of the package.

<3

@lheckemann
Copy link
Member

I'm inclined to agree with the approach of not keeping and serving debug symbols other than for packages like glibc. The missing piece for this, as far as I know, is support in hydra for only keeping a subset of outputs. Perhaps this would be best achieved by an extra meta field which defines outputs to discard or outputs to keep?

@layus
Copy link
Member

layus commented Jan 26, 2022

Discarding debug outputs seems the best way to go for now. Adding them to the cache for a fixed amount of time may come second, as a nice-to-have feature. So 👍 for me.

We should however keep in mind that it is not perfect. There is a risk of franken(debug)builds, namely the possibility that you associate a locally re-generated *-debug output with a cached output, and get mismatches because of build impurities. This would not happen if all the builds were perfectly reproducible, but that is not the case. See https://gist.github.com/layus/2a91694be330d5f2591dab9a52c47548 for a mwe.

Nix managed to avoid showing that issue by never garbage collecting its cache (!), and always uploading the full closure and outputs set of any cached path.

@lheckemann
Copy link
Member

That's a very good point. I think (hope) that it would be rare enough for debug symbols to be incompatible with the binaries that this isn't a major issue, but if it ever does happen we need a way to get a derivation rebuilt with the outputs replacing the previous versions... and I'm not sure what the best ergonomics for that would be. I have three ideas currently:

  • Alter nix-store --repair-path so that it will rebuild the path even if it's not corrupted; this would allow solving the problem with nix-store --repair-path $path --substituters ''. This is the closest we already have to this functionality, AFAIK;
  • A new flag for nix-build and nix-store -r which would behave sort of like --check (in that it rebuilds the path even if it's already there or if it can be substituted) and sort of like --repair-path (replacing what's currently in the store);
  • A new command, nix rebuild.

I'm not very happy with any of these, if anyone has any good ideas I'd appreciate them!

@Gaelan
Copy link
Contributor

Gaelan commented Jan 27, 2022

There's a simpler solution here (I think): modify Nix to never include two outputs from different sources in the store. So if you have foo-bin in the store, and you ask for foo-debug, Nix either:

  • gets foo-debug from the same substituter that it got foo-bin from (if this is a thing Nix knows); if we don't have this information, or if that substituter doesn't have foo-debug, we:
  • look for another substituter with both foo-bin and foo-debug, and download that; and if no such substituter exists, then:
  • build foo ourselves and put all the outputs into the store

@lheckemann
Copy link
Member

lheckemann commented Jan 27, 2022

Of course! While tracking provenance of paths is complicated (Nix doesn't currently do this), replacing all outputs with what's been built as in your last suggestion seems pretty feasible to me. Maybe Nix even already does this. Thanks :D

@lheckemann
Copy link
Member

Maybe Nix even already does this.

Unfortunately, it doesn't:

Script I used to test this
#!/usr/bin/env bash
set -exuo pipefail

export NIX_PATH=nixpkgs=/home/linus/nixpkgs/master
root=/scratch/missing-output-test
cachestore=$root/cache
deststore=$root/testing

for store in $deststore; do
    if [[ -d "$store" ]]; then
        chmod -R u+w $store
        rm -r $store
    fi
    mkdir -p $store
done

opts=(--secret-key-files '' --builders '')

cd $root
cat >expr.nix <<\EOF
let pkgs = import <nixpkgs> {}; in
rec {
  hello = pkgs.hello.overrideAttrs (o: {
    preConfigure = "exec 5>&2 2>/dev/null >/dev/null";
    separateDebugInfo = true;
    postFixup = "date | tee $out/date $debug/date";
  });
  refhello = pkgs.runCommand "refhello" {} ''
    date > $out
    echo ${hello} >> $out
  '';
}
EOF
drv=$(nix-instantiate "${opts[@]}" --store $cachestore expr.nix -A refhello)
out=$(nix-instantiate "${opts[@]}" --store $cachestore --eval expr.nix -A hello.out.outPath --json | jq -r)
debug=$(nix-instantiate "${opts[@]}" --store $cachestore --eval expr.nix -A hello.debug.outPath --json | jq -r)
ref=$(nix-instantiate "${opts[@]}" --store $cachestore --eval expr.nix -A refhello.outPath --json | jq -r)

nix-store "${opts[@]}" --store $cachestore -r $drv
nix-store "${opts[@]}" --store $cachestore --delete $debug

nix-build "${opts[@]}" --store $deststore --extra-substituters $cachestore --no-require-sigs expr.nix -A refhello
nix-build "${opts[@]}" --store $deststore --extra-substituters $cachestore --no-require-sigs expr.nix -A hello.debug

grep . {$cachestore,$deststore}/{{$out,$debug}/date,$ref}

I suspect it shouldn't be too hard to change that though.

@layus
Copy link
Member

layus commented Jan 28, 2022

There is no solution for that, it is a consequence of nix design. Replacing all the outputs just defers the issue to packages down the dependency chain. Also, as soon as you have multiple caches, or a cache starts dropping entries, you get this issue. Replacing all the outputs fixes inter-outputs provenance incompatibilities. But then all the packages that depended on the previous, replaced output may suffer from inconsistencies with the new one. Even though there should be even less inconsistencies between packages than between outputs of a single package.

I just wanted to make a note about this blind spot in the design of nix, that remains well hidden as long as we all rely on a shared central cache that does not discard any path. And discarding some paths is exactly the solution to the problem at hand here.

So let's leave a note here about this issues for people to find out if they ever encounter this, and let's move on.

Who has enough accesses to enable to make the required tests and changes ?

@edolstra
Copy link
Member

modify Nix to never include two outputs from different sources in the store.

This isn't going to happen because it's a pretty fundamental assumption in Nix that store paths with the same name are substitutable for each other (whether by actually substituting them from a binary cache, or by (re)building them locally). But this issue is moot as long as builds are binary-reproducible, which is what we should strive for anyway. (E.g. CAS Nix doesn't work as well if packages are not binary-reproducible.)

@lheckemann
Copy link
Member

Who has enough accesses to enable to make the required tests and changes ?

I suppose the next step would be hydra support for selectively copying outputs to the cache. Anyone could implement this :)

@Ericson2314
Copy link
Member

@layus I think with CA derivations this would be solved, because the debug output references the original, it will require it match a content-address; mixing output from non-deterministic builds won't work.

@layus
Copy link
Member

layus commented Feb 16, 2022

@Ericson2314 Yes indeed, because CA preserves the provenance relation ;-).

@Ericson2314
Copy link
Member

(C.F. #4344)

@Artturin Artturin modified the milestones: 21.05, 23.05 Dec 31, 2022
@vcunat
Copy link
Member

vcunat commented Jan 31, 2023

Debug info size: as we're about to default to gcc12 soon, it might be interesting to look into -gctf1 which should be way smaller than normal debug info (-g / -ggdb) and still provide backtraces. https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html

The format is just from the past few years though, so it's likely there will be some hurdles to solve first.

@Artturin Artturin removed this from the 23.05 milestone Feb 28, 2023
@symphorien
Copy link
Member

We should however keep in mind that it is not perfect. There is a risk of franken(debug)builds, namely the possibility that you associate a locally re-generated *-debug output with a cached output, and get mismatches because of build impurities. This would not happen if all the builds were perfectly reproducible, but that is not the case. See https://gist.github.com/layus/2a91694be330d5f2591dab9a52c47548 for a mwe.

this is already happening for qemu NixOS/nix#7756

@symphorien
Copy link
Member

NixOS/nix#8080 modifies nix so that it tags debug outputs when uploading them to s3 in a way that makes it possible to configure the s3 bucket backing cache.nixos.org to automatically remove debug outputs and debug outputs only after a configurable period of time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests