Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: introduce buildinfo files for reproducibility #2121

Open
wants to merge 2 commits into
base: master
from

Conversation

Projects
None yet
6 participants
@aparcar
Copy link
Member

commented Jun 13, 2019

generate feeds.buildinfo and commit.buildinfo in build dir after
containing the feed revisions (via ./scripts/feeds list -sf) as well as
the current revision of buildroot (via ./scripts/getver.sh).

With this information it should be possible to reproduce any build,
especially the release builds.

Usage would be to move feeds.buildinfo to feeds.conf and git checkout the
revision hash of commit.buildinfo.

Content of feeds.buildinfo would look similar to this:

src-git routing https://git.openwrt.org/feed/routing.git^bf475d6
src-git telephony https://git.openwrt.org/feed/telephony.git^470eb8e
...

Content of commit.buildinfo would look similar to this:

commit=r10203+1-c12bd3a21b

Without the exact feed revision it is not possible to determine
installed package versions.

Also rename config.seed to config.buildinfo to follow the recommended
style of https://reproducible-builds.org/docs/recording/

Signed-off-by: Paul Spooren mail@aparcar.org

@aparcar

This comment has been minimized.

Copy link
Member Author

commented Jun 13, 2019

Makefile Outdated
@@ -92,6 +92,8 @@ checksum: FORCE
diffconfig: FORCE
mkdir -p $(BIN_DIR)
$(SCRIPT_DIR)/diffconfig.sh > $(BIN_DIR)/config.seed
echo "# buildroot $$(git --git-dir $(TOPDIR)/.git rev-parse HEAD)" > $(BIN_DIR)/feeds.seed

This comment has been minimized.

Copy link
@jow-

jow- Jun 13, 2019

Contributor

Note that this may fail if buildroot does not reside in Git (which is something we do support at least for releases). This can for example happen if you download a release tarball from the Github release page - such a tarball will come without any metadata.

Maybe it would be better to use the output of ./scripts/getver.sh instead which is able to cope with such a situation.

This comment has been minimized.

Copy link
@ynezz

ynezz Jun 13, 2019

Member

From my POV it's not diffconfig target anymore, you're as well adding feeds config and build config which are probably separate things.

So this probably should be something like:

buildconfig: FORCE
    echo "commit=$$(git --git-dir $(TOPDIR)/.git rev-parse HEAD)" > $(BIN_DIR)/build.seed

feedsconfig: FORCE
   $(SCRIPT_DIR)/feeds list -fs >> $(BIN_DIR)/feeds.seed   

prepare: .config $(tools/stamp-compile) $(toolchain/stamp-compile)
	$(_SINGLE)$(SUBMAKE) -r diffconfig feedsconfig buildconfig

so you can then just source build.seed in some script/make target and simply use it, without any additional parsing. And I think, that it's likely, that we (or downstream projects) might need to add also other variables to the build.seed in order to provide enough information for reproducible builds.

This comment has been minimized.

Copy link
@aparcar

aparcar Jun 13, 2019

Author Member

Switched to using getver.sh and introduced buildconfig and feedsconfig.

@h01ger

This comment has been minimized.

Copy link

commented Jun 13, 2019

@aparcar aparcar changed the title build: introduce feeds.seed for reproducibility [RFC]build: introduce feeds.seed for reproducibility Jun 13, 2019

@aparcar aparcar force-pushed the aparcar:feeds-seed branch 3 times, most recently from cb0be9f to 74ee4f6 Jun 13, 2019

@aparcar aparcar changed the title [RFC]build: introduce feeds.seed for reproducibility [RFC]build: introduce buildinfo files for reproducibility Jun 13, 2019

@aparcar

This comment has been minimized.

Copy link
Member Author

commented Jun 13, 2019

Talking with @h01ger I decided to rename all files required for reproducibility to +.buildinfo as described here.

config.seed is now config.buildinfo, however that shouldn't bother to much as it's also changed in the make menuconfig.

@h01ger

This comment has been minimized.

Copy link

commented Jun 13, 2019

build: add buildinfo files for reproducibility
generate feeds.buildinfo and commit.buildinfo in build dir after
containing the feed revisions (via ./scripts/feeds list -sf) as well as
the current revision of buildroot (via ./scripts/getver.sh).

With this information it should be possible to reproduce any build,
especially the release builds.

Usage would be to move feeds.buildinfo to feeds.conf and git checkout the
revision hash of commit.buildinfo.

Content of feeds.buildinfo would look similar to this:

    src-git routing https://git.openwrt.org/feed/routing.git^bf475d6
    src-git telephony https://git.openwrt.org/feed/telephony.git^470eb8e
    ...

Content of commit.buildinfo would look similar to this:

    commit=r10203+1-c12bd3a21b

Without the exact feed revision it is not possible to determine
installed package versions.

Also rename config.seed to config.buildinfo to follow the recommended
style of https://reproducible-builds.org/docs/recording/

Signed-off-by: Paul Spooren <mail@aparcar.org>

@aparcar aparcar force-pushed the aparcar:feeds-seed branch from 74ee4f6 to 7cd8d15 Jun 13, 2019

@aparcar

This comment has been minimized.

Copy link
Member Author

commented Jun 15, 2019

I’d very much see this merged. Once merged I can create a rebuild tool to verify not only release builds but also user builds offered in the OpenWrt forum.

@aparcar aparcar changed the title [RFC]build: introduce buildinfo files for reproducibility build: introduce buildinfo files for reproducibility Jun 19, 2019

@aparcar

This comment has been minimized.

Copy link
Member Author

commented Jun 21, 2019

@lynxis ping 🦆

build: add rebuild.sh script
The rebuild.sh script allows to recreate images based on buildinfo
files. This automates the process to verify if provided buildinfo files
result in the expected images.

To run the files commit.buildinfo, feeds.buildinfo, config.buildinfo and
sha256sums are required in $BUILD_DIR (defaul ./rebuild).

Signed-off-by: Paul Spooren <mail@aparcar.org>
@aparcar

This comment has been minimized.

Copy link
Member Author

commented Jul 2, 2019

@h01ger could you please check the script I attached? Would be very happy this gets merged

@h01ger

This comment has been minimized.

Copy link

commented Jul 6, 2019

[ -z "$COMMIT" ] && exit 1

[ -d "$REBUILD_DIR/openwrt" ] || \
git clone https://github.com/openwrt/openwrt.git "$REBUILD_DIR/openwrt/" --depth 1

This comment has been minimized.

Copy link
@dangowrt

dangowrt Jul 17, 2019

Member

Couldn't the URL come from buildinfo was well instead of hard-coding it here? Then at least you'd know that a certain binary is based on a different tree (there are a lot of forks of OpenWrt, vendor SDKs, ...)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.