Reproducibility: Track external commands used #936
Comments
|
Excluding the |
|
(some of those are spurious -- |
|
@osresearch I am assuming commands executed within scripts would appear separately. Couldn't shell scripts be removed from this list to simplify it? Additionally, I am trying to compile Heads and coreboot in Guix. One of the features is the ability to generate deep dependency graphs. I'll post that here as soon as I am able to. |
|
That deep dependency graph is quite neat. I was thinking about making a text version of a dependency tree, although my immediate goal was a little simpler: ensure that we knew which external tools we were using to help identify where reproducibility issues might be creeping in. For instance, where are the 128 invocations of We also have an issue with #892 - And I'm especially amused by the 10 executions of |
|
@Thrilleratplay any update, blog post, pointers out to fix zlib problem outlined here and on slack guix-buildstack channel? |
|
@tlaurion Unfortunately, not really. I started down the path of extracting more and more Debian packages to be able to build crossgcc and it got to the point where it was obvious this defeated the purpose of using Guix. Ada released a new version a few weeks ago that may help overcome the bootstrapping issue. I haven't done much with it in the past month; I have been catching up with other obligations in different projects/life. Sorry. I'll try to get back to this soon. |
|
@tlaurion I had sometime last night to take a look at where I was with this. Basically, I keep going around in the same circle trying to resolve dependencies with Ada, Gnat, GCC and Zlib. A few projects have asked or seeked out Ada/Gnat for Guix (example 1 and the Guix IRC logs containing the word gnat). Based on what samplet says here, my problem may be that the AdaCore binaries are not static. This coupled with the fact that the last 32bit AdaCore release was in 2014 and using GCC 4.7.4, this may not be possible |
…and gawk are not found in expected 4.2.1 versions. This will be the case on debian-11 and other newer OSes. Fixes a bug where "make BOARD=xyz bootstrap" was needed prior of being able to build anything else otherwise HEADS_GAWK was expecting HEADS_MAKE to be built already, which was not the case. (fixes linuxboot#1126) As of now, I do not see any OS make/gawk being used anymore. All modules except coreboot (which uses coreboot buildstack), including musl-cross-make, are being built with local make and local gawk. Improves linuxboot#936 (comment) which showed that local make (/usb/bin/make, /usr/local/bin/make, /usr/sbin/make, /usr/local/sbin/make and /usr/bin/gawk were called locally. To redo)
…gawk locally built in expected 4.2.1 versions. This applies to debian-11 and other newer OSes. Fixes a bug where "make BOARD=xyz bootstrap" was needed prior of being able to build anything else otherwise HEADS_GAWK was expecting HEADS_MAKE to be built already, which was not the case. (fixes linuxboot#1126) As of now, I do not see any OS make/gawk being used anymore. All modules except coreboot (which uses coreboot buildstack), including musl-cross-make, are being built with local make and local gawk. Improves linuxboot#936 (comment) which showed that local make (/usb/bin/make, /usr/local/bin/make, /usr/sbin/make, /usr/local/sbin/make and /usr/bin/gawk were called locally. To redo)
…pected version. Maximized boards: have rom hashes outputted under hashes.txt. Still not clear why the Makefile "all" target is not doing that for each target. - Fixes a bug where "make BOARD=xyz bootstrap" was needed prior of being able to build anything else otherwise HEADS_GAWK was expecting HEADS_MAKE to be built already, which was not the case. (fixes linuxboot#1126) - As of now, I do not see any OS make/gawk being used anymore. All modules except coreboot (which uses coreboot buildstack), including musl-cross-make, are being built with local make and local gawk. Improves linuxboot#936 (comment) which showed that local make (/usb/bin/make, /usr/local/bin/make, /usr/sbin/make, /usr/local/sbin/make and /usr/bin/gawk were called locally. To redo)
|
Interesting enough, #1008 just proved that the issue is more important then known before. On same debian-11 docker image with required tools installed atop (per CircleCI), two builds of the same commit produced different busybox binaries 30 days apart. This kind of issue will be resolved when a static docker image, without installing additional tools, will be referred per Heads commit ID, pointing CircleCI to use that static docker image. When this is done, a same Heads commit should be able to build the same output (if downloaded tarballs are still accessible of course, which would be #1198 ). And of course, ideally, that docker image would be produced in a reproducible way, which would be #1191 but based on NixOS with a package list pointing to a static (non-live) commit, permitting to always download the same packages and reproducing the same docker image/the same build layer on top of a linux system when dropping under NixOS shell (this is https://nixos.org/guides/towards-reproducibility-pinning-nixpkgs.html) |
|
Long time lurker speaking up. I took a stab at using nix a few weeks ago and ran into ada issues and didn't pursue because it seemed guix was the intended direction. If thats not the case and nix is an option to be considered I'd be happy to give it a stab. If so I'd also go with nix flakes instead of non-flake'd-nix, flakes makes many things easier for projects trying to use nix imo (especially with direnv and devshell too). |
|
@mmlb new to nix myself here. The easiest would be to bump all boards to coreboot 4.17/4.18 and have newer buildstack bootstrap coreboot musl buildstack. While continuing to depend on old version of musl-cross-make to build everything not coreboot buildstack. Any challenge here is more then welcome, of course. No idea of the differences between nix flakes or nin-flake'd-nix here. But last time I checked, NixOS had Ada where guix didn't, and as of today, the nixos community and directions seems to make things more easy for Heads to use an OS as a builder more then integrating Heads into derivatives, which was attempted with Guix a while back as a PoC |
|
@mmlb this discussion thread might be helpful to understand the whys and determine the how osresearch/linux-builder#1 |
Hmm iirc the issue was with having gcc find the ada compiler. I either had cross-compiling enabled gcc or ada-enabled gcc but the cross gcc did not have ada. Not sure if that was the issue but I'll try to reproduce and get to the bottom of it.
Yep iirc thats what I was trying to go with too.
nix flakes is mostly nice here because it has well defined entry points in to the nix setup for the project and lots of tooling is being built up around that.
Yep most of my use of nix in projects is for a reproducible build environment. Lets aim for that first. |
|
@tlaurion I've started on something would a PR to here be the way to go or into linux-builder? |
|
@mmlb : how to use your branch? |
|
Hey @tlaurion idk why I didn't catch these notifications. I'll get a branch and draft PR up here then. It was initially pretty slow going because I kept hitting a gnat bootstrap issue in nixpkgs but I've been able to move past that. My next step was to go rtfm on circle ci to see how I can start testing my branch as the build env for CI, but I'll go ahead and just get the pr up so there's something to look at and maybe get feedback/help with circleci. |
|
A somewhat annoying but workable issue is that a bunch of the external projects that get built have hard coded paths to binaries that don't exist on NixOS or vanilla nix docker image ( |
|
@mmlb I quickly checked your commit master...mmlb:osresearch-heads:nix, including the patches you added for coreboot 4.13 for gnat that doesn't seem required per https://nixos.wiki/wiki/Coreboot Here is what I came up with, playing with nixos docker image and nix-shell trying to wrap what i've read in the past to start a PoC myself. The concept I got with nix-shell is that it permits you to make sure that binaries and libraries available are as defined. First steps here, so learning my way but:
Of course, I came across discrepencies as commented above. bash python and other paths are searched per shebang in scripts, seem like the better way to do so would be to have those shebang defined as Looked into flake a bit as well. Was wondering if necessary and if so why? nix-shell would not be enough? Steps I see:
Thoughts? |
|
I have no idea why I'm not seeing the pings from this repo :(, this is the only repo this happens on ✊...
Yep this would be fantastic but would take a lot of effort going through all the upstreams to get this fixed. Worthwhile for sure, but not immediate. We could probably make use of
The big benefit to flakes over nix-shell is that flakes provide known entrypoints and better tooling for known/expected uses. Things like updating nixpkgs pins and better integration with direnv. nix-shell is not necessarily out of the picture but I've enjoyed porting my usage of nix-shell over to flakes. That being said I'm going to try out the shell.nix example you linked to see how it compares with my flake (the
I was wondering if we could bypass musl-cross-make alltogether. It seems silly to use a build tool to build another build tool when the first one could be used alone. But I'd leave that for after we get nix/nixpkgs building the current setup. |
Running
strace -f -e execve makeallows us to see what external programs are used and how many times in the build process. If we want to ensure that only trusted/reproducible tools are used, then we might want to build or depend on our own versions of these.The text was updated successfully, but these errors were encountered: