Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure identical deterministic build results and update release process #1549

Closed
ageis opened this issue Oct 17, 2016 · 23 comments
Closed

Ensure identical deterministic build results and update release process #1549

ageis opened this issue Oct 17, 2016 · 23 comments
Labels
Milestone

Comments

@ageis
Copy link
Contributor

@ageis ageis commented Oct 17, 2016

#1521 and #1541 added deterministic builds with Gitian.

@str4d and @bitcartel compared manifests, and there was a diff in cache_package_manifest (the depends).

This ticket is open to ensure that we run the Gitian builds using the tagged rc1 release, generate the same binaries, push our signatures to gitian.sigs and then we update the release process document for the Gitian builds and binaries for rc2.

@daira daira added this to the 1.0.0-rc2 milestone Oct 18, 2016
@daira daira added the A-build label Oct 18, 2016
@ageis
Copy link
Contributor Author

@ageis ageis commented Oct 18, 2016

👍 I know I've been harping on this, but if we're serious about this, I'd say we need at least three devs publishing the build.assert and sig per release.

@radix42
Copy link
Contributor

@radix42 radix42 commented Oct 19, 2016

Do you guys want someone who isn't employed by zcashco to do and sign gitian builds for your releases, too? I'm assuming that the answer to that would be a "yes"....I could spin up a system on AWS that's big enough to do so in a reasonable amount of time that I only fire up for tagged releases.

@daira
Copy link
Contributor

@daira daira commented Oct 19, 2016

@radix42 Yes please!

@radix42
Copy link
Contributor

@radix42 radix42 commented Oct 19, 2016

I haven't run vagrant or gitian before, so it might take a while for me to get it all setup

@ageis
Copy link
Contributor Author

@ageis ageis commented Oct 19, 2016

@radix42 absolutely. We've automated a particular build environment setup in our code, so it's not a lot of work for you other than installing prerequisites and running a couple commands -- less than 30 min until you're running the Gitian builds, even. If you want to use other virtualization platforms with gitian-builder or end up running into trouble with VirtualBox on AWS, you're kinda on your own, however contact me if you have specific questions. It's good if we have more diverse build environments (for example, using KVM or Vbox instead of LXC for the actual build container), but at a minimum they should be Debian jessie based.

@radix42
Copy link
Contributor

@radix42 radix42 commented Oct 19, 2016

@ageis I'm not feeling up to rebasing my Windows port onto rc1 at the moment, so I'll work on getting this setup in my AWS account. There are a few spots in https://github.com/zcash/zcash-gitian that are ambiguous....should I add any questions I have about any roadblocks I hit to this ticket or a new/different one?

@ageis
Copy link
Contributor Author

@ageis ageis commented Oct 19, 2016

@radix42 you're welcome to also message me on the community Slack if you like. We tried to make the README clear to follow. :)

1 similar comment
@ageis
Copy link
Contributor Author

@ageis ageis commented Oct 19, 2016

@radix42 you're welcome to also message me on the community Slack if you like. We tried to make the README clear to follow. :)

@ageis ageis closed this Oct 19, 2016
@ageis ageis reopened this Oct 19, 2016
@radix42
Copy link
Contributor

@radix42 radix42 commented Oct 19, 2016

I get different results when I do a gitian build of rc1 (6BE5B036 is my gpg key ID):

vagrant@zcash-build:~/gitian.sigs/v1.0.0-rc1$ diff 6BE5B036/zcash-1.0.0-rc1-build.assert jack@z.cash/zcash-1.0.0-rc1-build.assert
4,5c4,5
<     91d381342f8376dedc727015d16cdb4ec232a2cf4c3d97a9f1acae919127a7f2  zcash-1.0.0-rc1-linux64-debug.tar.gz
<     42e1041fa916cca4ab559bb37abf6e6324c62952bffc85df64be053e4eaca730  zcash-1.0.0-rc1-linux64.tar.gz
---
>     267bed0369f8add2b0962cac944390db7bba5d78b588898558ff21dc963eb352  zcash-1.0.0-rc1-linux64-debug.tar.gz
>     60c4a368d0010924e3c3f168d1328674603598629dd3822ffa06d32703f7e812  zcash-1.0.0-rc1-linux64.tar.gz
341,342c341,342
<     a0961ad8c419975d3094843f8fb37511740fee78417125afef0d3ad4eff38ce7  x86_64-unknown-linux-gnu/libsnark/libsnark-0.1-bd7eaedfe8d.tar.gz
<     9aecb774d8c36840bdd70a8b50a9b2ac448f4ad9bbd7181a6638a934ec44dcc2  x86_64-unknown-linux-gnu/libsnark/libsnark-0.1-bd7eaedfe8d.tar.gz.hash
---
>     5db162459687cd350eca3e6927c65e9ab34980a860b9f3dbaf19cbf09ef3b304  x86_64-unknown-linux-gnu/libsnark/libsnark-0.1-bd7eaedfe8d.tar.gz
>     a3dcb0ba421c678e364148aed86ca9d7354bab31f5bacf15aa9c75cc9d308992  x86_64-unknown-linux-gnu/libsnark/libsnark-0.1-bd7eaedfe8d.tar.gz.hash
@ageis
Copy link
Contributor Author

@ageis ageis commented Oct 19, 2016

@radix42 Can you please find and scp zcash-1.0.0-rc1-linux64.tar.gz + libsnark-0.1-bd7eaedfe8d.tar.gz files out of the VM and try to attach it here or on Slack.

@radix42
Copy link
Contributor

@radix42 radix42 commented Oct 19, 2016

@ageis Here they are. Also, my gitian.yml is below, can you see what I set wrong that makes my gpg ID show up rather than my email for my key, as you guys have on your sigs?
libsnark-0.1-bd7eaedfe8d.tar.gz
zcash-1.0.0-rc1-linux64.tar.gz

gitian.yml (it won't let you attach a yml file):

---
- name: Apply the Zcash Gitian builder role.
  become: yes
  hosts: localhost:zcash-build
  vars:
    zcash_git_repo_url: https://github.com/zcash/zcash
    zcash_version: v1.0.0-rc1
    gpg_key_name: '6BE5B036'
    git_name: 'David Mercer'
    git_email: 'radix42@gmail.com'
    gpg_key_id: '6BE5B036' # optional
    ssh_key_name: 'id_rsa' # optional
  roles:
    - role: common
      tags: common
    - role: gitian
      tags: gitian
@str4d
Copy link
Contributor

@str4d str4d commented Oct 19, 2016

Here's the diff of my and @radix42's libsnark builds:

libsnark-rc1-str4d-vs-radix42.diff.txt

@radix42
Copy link
Contributor

@radix42 radix42 commented Oct 19, 2016

@ageis @str4d Here's my build.log, perhaps a diff of that and one of yours will reveal what's up with libsnark in my gitian build of rc1
radix42-gitian-rc1-build.log.zip

@str4d
Copy link
Contributor

@str4d str4d commented Oct 19, 2016

@radix42
Copy link
Contributor

@radix42 radix42 commented Oct 19, 2016

I think I found the difference:

$ diff str4d/build.log radix42/build.log

1797c1797
< checking build system type... sandybridge-pc-linux-gnu
---
> checking build system type... westmere-pc-linux-gnu
@radix42
Copy link
Contributor

@radix42 radix42 commented Oct 19, 2016

@ageis @str4d I think if we pass in --build=x86_64-unknown-linux-gnu to ./configure in gitian, libsnark should always build the same....the host is being passed in via the HOST environment variable (so it seems from my reading of what's going on), but it is figuring out the most specific processor it can generate code for because we aren't explicitly setting build, and this seems to make a difference when building libsnark. We'll want to do this, as per https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html the libsnark binary being built right now by gitian may or may not even RUN on a processor with a different specific instruction set (its optimizing too much!)

@radix42
Copy link
Contributor

@radix42 radix42 commented Oct 19, 2016

I myself would consider this a blocker until its fixed

@radix42
Copy link
Contributor

@radix42 radix42 commented Oct 19, 2016

I'm making a branch in my fork, I'm pretty sure we just need to pass in --host and --build to ./configure in build.sh explicitly, and that should make gitian builds the same regardless of the underlying processor architecture (provided of course that it is some flavor of x86_64).

@radix42
Copy link
Contributor

@radix42 radix42 commented Oct 20, 2016

Testing a further (one character!) patch to depends/Makefile so it respects both HOST and BUILD options passed in by build.sh. As soon as I have results from this gitian build, I'll submit a PR to upstream (you guys), assuming this works. It turns out that it is libgmp that is detecting the underlying processor instruction set at a finer level of detail than typical for gcc builds, and THAT is causing the differences in libsnark.

@radix42
Copy link
Contributor

@radix42 radix42 commented Oct 20, 2016

My pull request 1577 should remove the remaining indeterminism from v1.0.0-rc1 builds in libgmp and libsnark

@radix42
Copy link
Contributor

@radix42 radix42 commented Oct 20, 2016

I just verified that PR 1577 does indeed remove processor/chipset architecture from the build....of course you'll want to run it through the full test suite.

@daira
Copy link
Contributor

@daira daira commented Oct 20, 2016

Assuming fixed by #1577.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.