build: introduce buildinfo files for reproducibility #2121

aparcar · 2019-06-13T00:32:29Z

generate feeds.buildinfo and commit.buildinfo in build dir after
containing the feed revisions (via ./scripts/feeds list -sf) as well as
the current revision of buildroot (via ./scripts/getver.sh).

With this information it should be possible to reproduce any build,
especially the release builds.

Usage would be to move feeds.buildinfo to feeds.conf and git checkout the
revision hash of commit.buildinfo.

Content of feeds.buildinfo would look similar to this:

src-git routing https://git.openwrt.org/feed/routing.git^bf475d6
src-git telephony https://git.openwrt.org/feed/telephony.git^470eb8e
...

Content of commit.buildinfo would look similar to this:

commit=r10203+1-c12bd3a21b

Without the exact feed revision it is not possible to determine
installed package versions.

Also rename config.seed to config.buildinfo to follow the recommended
style of https://reproducible-builds.org/docs/recording/

Signed-off-by: Paul Spooren mail@aparcar.org

aparcar · 2019-06-13T00:41:19Z

@h01ger @lynxis

jow- · 2019-06-13T06:23:13Z

Makefile

@@ -92,6 +92,8 @@ checksum: FORCE
 diffconfig: FORCE
 	mkdir -p $(BIN_DIR)
 	$(SCRIPT_DIR)/diffconfig.sh > $(BIN_DIR)/config.seed
+	echo "# buildroot $$(git --git-dir $(TOPDIR)/.git rev-parse HEAD)" > $(BIN_DIR)/feeds.seed


Note that this may fail if buildroot does not reside in Git (which is something we do support at least for releases). This can for example happen if you download a release tarball from the Github release page - such a tarball will come without any metadata.

Maybe it would be better to use the output of ./scripts/getver.sh instead which is able to cope with such a situation.

From my POV it's not diffconfig target anymore, you're as well adding feeds config and build config which are probably separate things.

So this probably should be something like:

buildconfig: FORCE echo "commit=$$(git --git-dir $(TOPDIR)/.git rev-parse HEAD)" > $(BIN_DIR)/build.seed feedsconfig: FORCE $(SCRIPT_DIR)/feeds list -fs >> $(BIN_DIR)/feeds.seed prepare: .config $(tools/stamp-compile) $(toolchain/stamp-compile) $(_SINGLE)$(SUBMAKE) -r diffconfig feedsconfig buildconfig

so you can then just source build.seed in some script/make target and simply use it, without any additional parsing. And I think, that it's likely, that we (or downstream projects) might need to add also other variables to the build.seed in order to provide enough information for reproducible builds.

Switched to using getver.sh and introduced buildconfig and feedsconfig.

h01ger · 2019-06-13T08:23:30Z

looks good for a start, however jow is right, you also need to support non-git checkouts... (and i'm a bit bothered/sad that you don't reuse the .buildinfo term.)

…

-- tschau, Holger

------------------------------------------------------------------------------- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

aparcar · 2019-06-13T14:49:19Z

Talking with @h01ger I decided to rename all files required for reproducibility to +.buildinfo as described here.

config.seed is now config.buildinfo, however that shouldn't bother to much as it's also changed in the make menuconfig.

h01ger · 2019-06-13T15:55:31Z

On Thu, Jun 13, 2019 at 07:49:49AM -0700, Paul Spooren wrote: Talking with @h01ger I decided to rename all files required for reproducibility to +.buildinfo as described [here](https://reproducible-builds.org/docs/recording/).

yay & thank you!

…

-- tschau, Holger

------------------------------------------------------------------------------- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C Dance like no one's watching. Encrypt like everyone is.

aparcar · 2019-06-15T17:15:20Z

I’d very much see this merged. Once merged I can create a rebuild tool to verify not only release builds but also user builds offered in the OpenWrt forum.

aparcar · 2019-06-21T20:16:49Z

@lynxis ping 🦆

aparcar · 2019-07-02T11:39:59Z

@h01ger could you please check the script I attached? Would be very happy this gets merged

h01ger · 2019-07-06T12:30:31Z

On Tue, Jul 02, 2019 at 04:40:23AM -0700, Paul Spooren wrote: @h01ger could you please check the script I attached?

looks good to me, though obviously i'm not an openwrt expert :)

Would be very happy this gets merged

same here.

…

-- tschau, Holger

------------------------------------------------------------------------------- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

dangowrt · 2019-07-17T12:53:52Z

scripts/rebuild.sh

+[ -z "$COMMIT" ] && exit 1
+
+[ -d "$REBUILD_DIR/openwrt" ] || \
+    git clone https://github.com/openwrt/openwrt.git "$REBUILD_DIR/openwrt/" --depth 1


Couldn't the URL come from buildinfo was well instead of hard-coding it here? Then at least you'd know that a certain binary is based on a different tree (there are a lot of forks of OpenWrt, vendor SDKs, ...)

And wouldn't be https://git.openwrt.org/openwrt/openwrt.git better anyway?

@dangowrt the rebuild.sh is just a "proof of concept", to be later used in a CI or something. However, it seems like a good idea to store the origin source in buildfiles. What would you suggest?

aparcar · 2019-07-31T13:18:52Z

The rebuild.sh is more of a proof of concept but a tool to be really added. As this PR is cherry-picked anyway, only the first commit should be picked. I'd be happy to see some progress here, once done I'd work together with @h01ger to integrate in in their CI.

h01ger · 2019-07-31T13:44:45Z

On Wed, Jul 31, 2019 at 06:19:16AM -0700, Paul Spooren wrote: The rebuild.sh is more of a *proof of concept* but a tool to be really added. As this PR is cherry-picked anyway, only the first commit should be picked.

then split this commit in two?

I'd be happy to see some progress here, once done I'd work together with @h01ger to integrate in in their CI.

sure!

…

-- tschau, Holger

------------------------------------------------------------------------------- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C we'll all die. make a difference while you can. disobey. smile.

aparcar · 2019-07-31T13:51:57Z

then split this commit in two?

Done

generate feeds.buildinfo and version.buildinfo in build dir after containing the feed revisions (via ./scripts/feeds list -sf) as well as the current revision of buildroot (via ./scripts/getver.sh). With this information it should be possible to reproduce any build, especially the release builds. Usage would be to move feeds.buildinfo to feeds.conf and git checkout the revision hash of version.buildinfo. Content of feeds.buildinfo would look similar to this: src-git routing https://git.openwrt.org/feed/routing.git^bf475d6 src-git telephony https://git.openwrt.org/feed/telephony.git^470eb8e ... Content of version.buildinfo would look similar to this: r10203+1-c12bd3a21b Without the exact feed revision it is not possible to determine installed package versions. Also rename config.seed to config.buildinfo to follow the recommended style of https://reproducible-builds.org/docs/recording/ Signed-off-by: Paul Spooren <mail@aparcar.org>

dangowrt · 2019-08-22T11:19:04Z

Cherry-picked onto master (a while ago).

jow- reviewed Jun 13, 2019

View reviewed changes

aparcar changed the title ~~build: introduce feeds.seed for reproducibility~~ [RFC]build: introduce feeds.seed for reproducibility Jun 13, 2019

aparcar force-pushed the feeds-seed branch 3 times, most recently from cb0be9f to 74ee4f6 Compare June 13, 2019 14:47

aparcar changed the title ~~[RFC]build: introduce feeds.seed for reproducibility~~ [RFC]build: introduce buildinfo files for reproducibility Jun 13, 2019

aparcar force-pushed the feeds-seed branch from 74ee4f6 to 7cd8d15 Compare June 13, 2019 15:59

dedeckeh added RFC pull request ready for comments build/scripts/tools pull request/issues for build, scripts and tools related changes labels Jun 15, 2019

aparcar changed the title ~~[RFC]build: introduce buildinfo files for reproducibility~~ build: introduce buildinfo files for reproducibility Jun 19, 2019

dangowrt reviewed Jul 17, 2019

View reviewed changes

aparcar force-pushed the feeds-seed branch from b1a113a to 7cd8d15 Compare July 31, 2019 13:51

aparcar force-pushed the feeds-seed branch from 7cd8d15 to a545945 Compare August 13, 2019 08:35

aparcar force-pushed the feeds-seed branch from a545945 to 68063fc Compare August 13, 2019 08:36

dangowrt closed this Aug 22, 2019

aparcar deleted the feeds-seed branch June 19, 2021 23:26

build: introduce buildinfo files for reproducibility #2121

build: introduce buildinfo files for reproducibility #2121

aparcar commented Jun 13, 2019 •

edited

aparcar commented Jun 13, 2019

jow- Jun 13, 2019

ynezz Jun 13, 2019

aparcar Jun 13, 2019

h01ger commented Jun 13, 2019 via email

aparcar commented Jun 13, 2019

h01ger commented Jun 13, 2019 via email

aparcar commented Jun 15, 2019

aparcar commented Jun 21, 2019

aparcar commented Jul 2, 2019

h01ger commented Jul 6, 2019 via email

dangowrt Jul 17, 2019

adschm Jul 23, 2019

aparcar Jul 31, 2019

aparcar commented Jul 31, 2019

h01ger commented Jul 31, 2019 via email

aparcar commented Jul 31, 2019

dangowrt commented Aug 22, 2019

build: introduce buildinfo files for reproducibility #2121

build: introduce buildinfo files for reproducibility #2121

Conversation

aparcar commented Jun 13, 2019 • edited

aparcar commented Jun 13, 2019

jow- Jun 13, 2019

Choose a reason for hiding this comment

ynezz Jun 13, 2019

Choose a reason for hiding this comment

aparcar Jun 13, 2019

Choose a reason for hiding this comment

h01ger commented Jun 13, 2019 via email

aparcar commented Jun 13, 2019

h01ger commented Jun 13, 2019 via email

aparcar commented Jun 15, 2019

aparcar commented Jun 21, 2019

aparcar commented Jul 2, 2019

h01ger commented Jul 6, 2019 via email

dangowrt Jul 17, 2019

Choose a reason for hiding this comment

adschm Jul 23, 2019

Choose a reason for hiding this comment

aparcar Jul 31, 2019

Choose a reason for hiding this comment

aparcar commented Jul 31, 2019

h01ger commented Jul 31, 2019 via email

aparcar commented Jul 31, 2019

dangowrt commented Aug 22, 2019

aparcar commented Jun 13, 2019 •

edited