Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

R4R: Infrastructure for reproducible builds #4262

Merged
merged 73 commits into from May 13, 2019

Conversation

@alessio
Copy link
Member

commented May 2, 2019

This change set introduces support for building gaia with gitian
on the following GOOS/GOARCH pairs:

  • darwin/386
  • darwin/amd64
  • linux/386
  • linux/amd64
  • linux/arm
  • linux/arm64
  • windows/386
  • windows/amd64

cmd/gaia/contrib/gitian-descriptors/ contains gitian descriptor files.

cmd/gaia/contrib/gitian-keys/ contains:

  • a keys.txt file that is meant to list core developers and gitian builders PGP keys.
  • README.me to provide instructions on how to import the keys into one's
    personal GPG keyring.

The gosum utility is removed, so is the go.sum hashsum bit from gaiacli/gaiad
version string. It was meant to be a provisional mitigation to the lack of a
reproducible build process.

GOBIN is removed from all Makefiles. When GOBIN is set, go refuses to cross-compiles
binaries for foreign architectures. export GOBIN=$GOPATH/bin is unnecessary anyway
as by default go install places built binaries in $GOPATH/bin. Developers are required
to update their enviornment files and replace $GOBIN with $GOPATH/bin in PATH.

circleci configuration file is amended accordingly.

Instructions at https://github.com/cosmos/cosmos-sdk/wiki/Reproducible-builds

Closes: #4027
Closes: #4280


For Admin Use:

  • Added appropriate labels to PR (ex. wip, ready-for-review, docs)
  • Reviewers Assigned
  • Squashed all commits, uses message "Merge pull request #XYZ: [title]" (coding standards)
@codecov

This comment has been minimized.

Copy link

commented May 3, 2019

Codecov Report

Merging #4262 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #4262   +/-   ##
======================================
  Coverage    59.1%   59.1%           
======================================
  Files         217     217           
  Lines       14595   14595           
======================================
  Hits         8627    8627           
  Misses       5330    5330           
  Partials      638     638

alessio added some commits May 2, 2019

Add Linux descriptor for Gitian builds
Instructions follow:

Make sure ruby is installed in your system and lives in /usr/bin/ruby:

 [ `which ruby` = '/usr/bin/ruby' ] && echo yes || echo no

Clone gitian:

 git clone https://github.com/devrandom/gitian-builder

Prepare a docker image for the build:

 cd gitian-build
 bin/make-base-vm --docker
 bin/make-base-vm --docker --arch amd64
 export USE_DOCKER=1

Download go:

  mkdir inputs
  gopkg=go1.12.4.linux-amd64.tar.gz ; \
  curl -L https://dl.google.com/go/$gopkg > inputs/$gopkg

Build:

  bin/gbuild $SDK-PATH/cmd/gaia/contrib/gitian-linux.yml \
    --commit cosmos-sdk=v0.34.3
Kill gosum
Update version package accordingly.

@alessio alessio force-pushed the alessio/reproducible-builds branch from 2022bb9 to 9f28890 May 3, 2019

alessio added some commits May 3, 2019

Revert "Set a different gopath"
This reverts commit e245095.

@alessio alessio changed the title Add Linux descriptor for Gitian builds Start building the infrastructure for reproducible builds May 3, 2019

@alessio alessio marked this pull request as ready for review May 3, 2019

@alessio alessio requested review from mircea-c and zmanian May 3, 2019

alessio added some commits May 9, 2019

Show resolved Hide resolved cmd/gaia/Makefile

alessio added some commits May 9, 2019

@alessio

This comment has been minimized.

Copy link
Member Author

commented May 9, 2019

Bnaries are now built with CGO enabled. I've successfully tested ledger support on Linux amd64.

@Liamsi
Copy link
Member

left a comment

If I run ./cmd/gaia/contrib/gitian-build.sh on macosx, I get a ./cmd/gaia/contrib/gitian-build.sh: line 196: 1: unbound variable. Update: OK, I see this needs a param passed in, e.g. ./cmd/gaia/contrib/gitian-build.sh linux pwd.

Now:

Start the build
bin/gbuild:252:in ``': No such file or directory - sha256sum (Errno::ENOENT)
        from bin/gbuild:252:in `<main>'

OK, looks like coreutils installation failed. NVM.

@Liamsi

This comment has been minimized.

Copy link
Member

commented May 10, 2019

The instructions should probably contain a sentence on how to verify that the build I've created matches the official build / I successfully reproduced it.

@alessio

This comment has been minimized.

Copy link
Member Author

commented May 10, 2019

@Liamsi dixit:

The instructions should probably contain a sentence on how to verify that the build I've created matches the official build / I successfully reproduced it.

I will certainly do! And I will include instructions on how to retrieve and import builder keys into your GPG keyring, etc.

@Liamsi

This comment has been minimized.

Copy link
Member

commented May 10, 2019

Update: coreutils wasn't in PATH ... NVM

I ran git pull to get the latest changes (and deleted the gaia.sigs and the gitian-build-linux dirs, and restarted the build (via ./cmd/gaia/contrib/gitian-build.sh -s Ismail.Khoffi@gmail.com linux `pwd`) and got:

Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Upgrading system, may take a while (log in var/install.log)
Creating package manifest
Creating build script (var/build-script)
date: illegal time format
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ... 
            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
bin/gbuild:147:in `block (2 levels) in build_one_configuration': error looking up author date in cosmos-sdk (RuntimeError)
        from bin/gbuild:143:in `each'
        from bin/gbuild:143:in `block in build_one_configuration'
        from bin/gbuild:127:in `open'
        from bin/gbuild:127:in `build_one_configuration'
        from bin/gbuild:309:in `block (2 levels) in <main>'
        from bin/gbuild:304:in `each'
        from bin/gbuild:304:in `block in <main>'
        from bin/gbuild:302:in `each'
        from bin/gbuild:302:in `<main>'
@Liamsi

This comment has been minimized.

Copy link
Member

commented May 10, 2019

alessio added some commits May 11, 2019

@alessio alessio requested a review from Liamsi May 13, 2019

@jackzampolin
Copy link
Contributor

left a comment

Builds the darwin for me!!

@alessio alessio requested a review from cwgoes May 13, 2019

@alessio alessio merged commit c0486aa into master May 13, 2019

15 checks passed

GolangCI No issues found!
Details
ci/circleci: docker_image Your tests passed on CircleCI!
Details
ci/circleci: integration_tests Your tests passed on CircleCI!
Details
ci/circleci: lint Your tests passed on CircleCI!
Details
ci/circleci: localnet Your tests passed on CircleCI!
Details
ci/circleci: setup_dependencies Your tests passed on CircleCI!
Details
ci/circleci: test_cover Your tests passed on CircleCI!
Details
ci/circleci: test_sim_gaia_fast Your tests passed on CircleCI!
Details
ci/circleci: test_sim_gaia_import_export Your tests passed on CircleCI!
Details
ci/circleci: test_sim_gaia_multi_seed Your tests passed on CircleCI!
Details
ci/circleci: test_sim_gaia_nondeterminism Your tests passed on CircleCI!
Details
ci/circleci: test_sim_gaia_simulation_after_import Your tests passed on CircleCI!
Details
ci/circleci: upload_coverage Your tests passed on CircleCI!
Details
codecov/patch Coverage not affected when comparing 2d3a251...9210caf
Details
codecov/project 59.38% (+0.01%) compared to 2d3a251
Details

@alessio alessio deleted the alessio/reproducible-builds branch May 13, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.