Ensure identical deterministic build results and update release process #1549

Closed
ageis opened this Issue Oct 17, 2016 · 23 comments

Comments

Projects
None yet
4 participants
@ageis
Contributor

ageis commented Oct 17, 2016

#1521 and #1541 added deterministic builds with Gitian.

@str4d and @bitcartel compared manifests, and there was a diff in cache_package_manifest (the depends).

This ticket is open to ensure that we run the Gitian builds using the tagged rc1 release, generate the same binaries, push our signatures to gitian.sigs and then we update the release process document for the Gitian builds and binaries for rc2.

@daira daira added this to the 1.0.0-rc2 milestone Oct 18, 2016

@daira daira added the build label Oct 18, 2016

@ageis

This comment has been minimized.

Show comment
Hide comment
@ageis

ageis Oct 18, 2016

Contributor

👍 I know I've been harping on this, but if we're serious about this, I'd say we need at least three devs publishing the build.assert and sig per release.

Contributor

ageis commented Oct 18, 2016

👍 I know I've been harping on this, but if we're serious about this, I'd say we need at least three devs publishing the build.assert and sig per release.

@radix42

This comment has been minimized.

Show comment
Hide comment
@radix42

radix42 Oct 19, 2016

Contributor

Do you guys want someone who isn't employed by zcashco to do and sign gitian builds for your releases, too? I'm assuming that the answer to that would be a "yes"....I could spin up a system on AWS that's big enough to do so in a reasonable amount of time that I only fire up for tagged releases.

Contributor

radix42 commented Oct 19, 2016

Do you guys want someone who isn't employed by zcashco to do and sign gitian builds for your releases, too? I'm assuming that the answer to that would be a "yes"....I could spin up a system on AWS that's big enough to do so in a reasonable amount of time that I only fire up for tagged releases.

@daira

This comment has been minimized.

Show comment
Hide comment
@daira

daira Oct 19, 2016

Contributor

@radix42 Yes please!

Contributor

daira commented Oct 19, 2016

@radix42 Yes please!

@radix42

This comment has been minimized.

Show comment
Hide comment
@radix42

radix42 Oct 19, 2016

Contributor

I haven't run vagrant or gitian before, so it might take a while for me to get it all setup

Contributor

radix42 commented Oct 19, 2016

I haven't run vagrant or gitian before, so it might take a while for me to get it all setup

@ageis

This comment has been minimized.

Show comment
Hide comment
@ageis

ageis Oct 19, 2016

Contributor

@radix42 absolutely. We've automated a particular build environment setup in our code, so it's not a lot of work for you other than installing prerequisites and running a couple commands -- less than 30 min until you're running the Gitian builds, even. If you want to use other virtualization platforms with gitian-builder or end up running into trouble with VirtualBox on AWS, you're kinda on your own, however contact me if you have specific questions. It's good if we have more diverse build environments (for example, using KVM or Vbox instead of LXC for the actual build container), but at a minimum they should be Debian jessie based.

Contributor

ageis commented Oct 19, 2016

@radix42 absolutely. We've automated a particular build environment setup in our code, so it's not a lot of work for you other than installing prerequisites and running a couple commands -- less than 30 min until you're running the Gitian builds, even. If you want to use other virtualization platforms with gitian-builder or end up running into trouble with VirtualBox on AWS, you're kinda on your own, however contact me if you have specific questions. It's good if we have more diverse build environments (for example, using KVM or Vbox instead of LXC for the actual build container), but at a minimum they should be Debian jessie based.

@radix42

This comment has been minimized.

Show comment
Hide comment
@radix42

radix42 Oct 19, 2016

Contributor

@ageis I'm not feeling up to rebasing my Windows port onto rc1 at the moment, so I'll work on getting this setup in my AWS account. There are a few spots in https://github.com/zcash/zcash-gitian that are ambiguous....should I add any questions I have about any roadblocks I hit to this ticket or a new/different one?

Contributor

radix42 commented Oct 19, 2016

@ageis I'm not feeling up to rebasing my Windows port onto rc1 at the moment, so I'll work on getting this setup in my AWS account. There are a few spots in https://github.com/zcash/zcash-gitian that are ambiguous....should I add any questions I have about any roadblocks I hit to this ticket or a new/different one?

@ageis

This comment has been minimized.

Show comment
Hide comment
@ageis

ageis Oct 19, 2016

Contributor

@radix42 you're welcome to also message me on the community Slack if you like. We tried to make the README clear to follow. :)

Contributor

ageis commented Oct 19, 2016

@radix42 you're welcome to also message me on the community Slack if you like. We tried to make the README clear to follow. :)

@ageis

This comment has been minimized.

Show comment
Hide comment
@ageis

ageis Oct 19, 2016

Contributor

@radix42 you're welcome to also message me on the community Slack if you like. We tried to make the README clear to follow. :)

Contributor

ageis commented Oct 19, 2016

@radix42 you're welcome to also message me on the community Slack if you like. We tried to make the README clear to follow. :)

@ageis ageis closed this Oct 19, 2016

@ageis ageis reopened this Oct 19, 2016

@radix42

This comment has been minimized.

Show comment
Hide comment
@radix42

radix42 Oct 19, 2016

Contributor

I get different results when I do a gitian build of rc1 (6BE5B036 is my gpg key ID):

vagrant@zcash-build:~/gitian.sigs/v1.0.0-rc1$ diff 6BE5B036/zcash-1.0.0-rc1-build.assert jack@z.cash/zcash-1.0.0-rc1-build.assert
4,5c4,5
<     91d381342f8376dedc727015d16cdb4ec232a2cf4c3d97a9f1acae919127a7f2  zcash-1.0.0-rc1-linux64-debug.tar.gz
<     42e1041fa916cca4ab559bb37abf6e6324c62952bffc85df64be053e4eaca730  zcash-1.0.0-rc1-linux64.tar.gz
---
>     267bed0369f8add2b0962cac944390db7bba5d78b588898558ff21dc963eb352  zcash-1.0.0-rc1-linux64-debug.tar.gz
>     60c4a368d0010924e3c3f168d1328674603598629dd3822ffa06d32703f7e812  zcash-1.0.0-rc1-linux64.tar.gz
341,342c341,342
<     a0961ad8c419975d3094843f8fb37511740fee78417125afef0d3ad4eff38ce7  x86_64-unknown-linux-gnu/libsnark/libsnark-0.1-bd7eaedfe8d.tar.gz
<     9aecb774d8c36840bdd70a8b50a9b2ac448f4ad9bbd7181a6638a934ec44dcc2  x86_64-unknown-linux-gnu/libsnark/libsnark-0.1-bd7eaedfe8d.tar.gz.hash
---
>     5db162459687cd350eca3e6927c65e9ab34980a860b9f3dbaf19cbf09ef3b304  x86_64-unknown-linux-gnu/libsnark/libsnark-0.1-bd7eaedfe8d.tar.gz
>     a3dcb0ba421c678e364148aed86ca9d7354bab31f5bacf15aa9c75cc9d308992  x86_64-unknown-linux-gnu/libsnark/libsnark-0.1-bd7eaedfe8d.tar.gz.hash
Contributor

radix42 commented Oct 19, 2016

I get different results when I do a gitian build of rc1 (6BE5B036 is my gpg key ID):

vagrant@zcash-build:~/gitian.sigs/v1.0.0-rc1$ diff 6BE5B036/zcash-1.0.0-rc1-build.assert jack@z.cash/zcash-1.0.0-rc1-build.assert
4,5c4,5
<     91d381342f8376dedc727015d16cdb4ec232a2cf4c3d97a9f1acae919127a7f2  zcash-1.0.0-rc1-linux64-debug.tar.gz
<     42e1041fa916cca4ab559bb37abf6e6324c62952bffc85df64be053e4eaca730  zcash-1.0.0-rc1-linux64.tar.gz
---
>     267bed0369f8add2b0962cac944390db7bba5d78b588898558ff21dc963eb352  zcash-1.0.0-rc1-linux64-debug.tar.gz
>     60c4a368d0010924e3c3f168d1328674603598629dd3822ffa06d32703f7e812  zcash-1.0.0-rc1-linux64.tar.gz
341,342c341,342
<     a0961ad8c419975d3094843f8fb37511740fee78417125afef0d3ad4eff38ce7  x86_64-unknown-linux-gnu/libsnark/libsnark-0.1-bd7eaedfe8d.tar.gz
<     9aecb774d8c36840bdd70a8b50a9b2ac448f4ad9bbd7181a6638a934ec44dcc2  x86_64-unknown-linux-gnu/libsnark/libsnark-0.1-bd7eaedfe8d.tar.gz.hash
---
>     5db162459687cd350eca3e6927c65e9ab34980a860b9f3dbaf19cbf09ef3b304  x86_64-unknown-linux-gnu/libsnark/libsnark-0.1-bd7eaedfe8d.tar.gz
>     a3dcb0ba421c678e364148aed86ca9d7354bab31f5bacf15aa9c75cc9d308992  x86_64-unknown-linux-gnu/libsnark/libsnark-0.1-bd7eaedfe8d.tar.gz.hash
@ageis

This comment has been minimized.

Show comment
Hide comment
@ageis

ageis Oct 19, 2016

Contributor

@radix42 Can you please find and scp zcash-1.0.0-rc1-linux64.tar.gz + libsnark-0.1-bd7eaedfe8d.tar.gz files out of the VM and try to attach it here or on Slack.

Contributor

ageis commented Oct 19, 2016

@radix42 Can you please find and scp zcash-1.0.0-rc1-linux64.tar.gz + libsnark-0.1-bd7eaedfe8d.tar.gz files out of the VM and try to attach it here or on Slack.

@radix42

This comment has been minimized.

Show comment
Hide comment
@radix42

radix42 Oct 19, 2016

Contributor

@ageis Here they are. Also, my gitian.yml is below, can you see what I set wrong that makes my gpg ID show up rather than my email for my key, as you guys have on your sigs?
libsnark-0.1-bd7eaedfe8d.tar.gz
zcash-1.0.0-rc1-linux64.tar.gz

gitian.yml (it won't let you attach a yml file):

---
- name: Apply the Zcash Gitian builder role.
  become: yes
  hosts: localhost:zcash-build
  vars:
    zcash_git_repo_url: https://github.com/zcash/zcash
    zcash_version: v1.0.0-rc1
    gpg_key_name: '6BE5B036'
    git_name: 'David Mercer'
    git_email: 'radix42@gmail.com'
    gpg_key_id: '6BE5B036' # optional
    ssh_key_name: 'id_rsa' # optional
  roles:
    - role: common
      tags: common
    - role: gitian
      tags: gitian
Contributor

radix42 commented Oct 19, 2016

@ageis Here they are. Also, my gitian.yml is below, can you see what I set wrong that makes my gpg ID show up rather than my email for my key, as you guys have on your sigs?
libsnark-0.1-bd7eaedfe8d.tar.gz
zcash-1.0.0-rc1-linux64.tar.gz

gitian.yml (it won't let you attach a yml file):

---
- name: Apply the Zcash Gitian builder role.
  become: yes
  hosts: localhost:zcash-build
  vars:
    zcash_git_repo_url: https://github.com/zcash/zcash
    zcash_version: v1.0.0-rc1
    gpg_key_name: '6BE5B036'
    git_name: 'David Mercer'
    git_email: 'radix42@gmail.com'
    gpg_key_id: '6BE5B036' # optional
    ssh_key_name: 'id_rsa' # optional
  roles:
    - role: common
      tags: common
    - role: gitian
      tags: gitian
@str4d

This comment has been minimized.

Show comment
Hide comment
@str4d

str4d Oct 19, 2016

Contributor

Here's the diff of my and @radix42's libsnark builds:

libsnark-rc1-str4d-vs-radix42.diff.txt

Contributor

str4d commented Oct 19, 2016

Here's the diff of my and @radix42's libsnark builds:

libsnark-rc1-str4d-vs-radix42.diff.txt

@radix42

This comment has been minimized.

Show comment
Hide comment
@radix42

radix42 Oct 19, 2016

Contributor

@ageis @str4d Here's my build.log, perhaps a diff of that and one of yours will reveal what's up with libsnark in my gitian build of rc1
radix42-gitian-rc1-build.log.zip

Contributor

radix42 commented Oct 19, 2016

@ageis @str4d Here's my build.log, perhaps a diff of that and one of yours will reveal what's up with libsnark in my gitian build of rc1
radix42-gitian-rc1-build.log.zip

@str4d

This comment has been minimized.

Show comment
Hide comment
Contributor

str4d commented Oct 19, 2016

@radix42

This comment has been minimized.

Show comment
Hide comment
@radix42

radix42 Oct 19, 2016

Contributor

I think I found the difference:

$ diff str4d/build.log radix42/build.log

1797c1797
< checking build system type... sandybridge-pc-linux-gnu
---
> checking build system type... westmere-pc-linux-gnu
Contributor

radix42 commented Oct 19, 2016

I think I found the difference:

$ diff str4d/build.log radix42/build.log

1797c1797
< checking build system type... sandybridge-pc-linux-gnu
---
> checking build system type... westmere-pc-linux-gnu
@radix42

This comment has been minimized.

Show comment
Hide comment
@radix42

radix42 Oct 19, 2016

Contributor

@ageis @str4d I think if we pass in --build=x86_64-unknown-linux-gnu to ./configure in gitian, libsnark should always build the same....the host is being passed in via the HOST environment variable (so it seems from my reading of what's going on), but it is figuring out the most specific processor it can generate code for because we aren't explicitly setting build, and this seems to make a difference when building libsnark. We'll want to do this, as per https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html the libsnark binary being built right now by gitian may or may not even RUN on a processor with a different specific instruction set (its optimizing too much!)

Contributor

radix42 commented Oct 19, 2016

@ageis @str4d I think if we pass in --build=x86_64-unknown-linux-gnu to ./configure in gitian, libsnark should always build the same....the host is being passed in via the HOST environment variable (so it seems from my reading of what's going on), but it is figuring out the most specific processor it can generate code for because we aren't explicitly setting build, and this seems to make a difference when building libsnark. We'll want to do this, as per https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html the libsnark binary being built right now by gitian may or may not even RUN on a processor with a different specific instruction set (its optimizing too much!)

@radix42

This comment has been minimized.

Show comment
Hide comment
@radix42

radix42 Oct 19, 2016

Contributor

I myself would consider this a blocker until its fixed

Contributor

radix42 commented Oct 19, 2016

I myself would consider this a blocker until its fixed

@radix42

This comment has been minimized.

Show comment
Hide comment
@radix42

radix42 Oct 19, 2016

Contributor

I'm making a branch in my fork, I'm pretty sure we just need to pass in --host and --build to ./configure in build.sh explicitly, and that should make gitian builds the same regardless of the underlying processor architecture (provided of course that it is some flavor of x86_64).

Contributor

radix42 commented Oct 19, 2016

I'm making a branch in my fork, I'm pretty sure we just need to pass in --host and --build to ./configure in build.sh explicitly, and that should make gitian builds the same regardless of the underlying processor architecture (provided of course that it is some flavor of x86_64).

@radix42

This comment has been minimized.

Show comment
Hide comment
@radix42

radix42 Oct 20, 2016

Contributor

Testing a further (one character!) patch to depends/Makefile so it respects both HOST and BUILD options passed in by build.sh. As soon as I have results from this gitian build, I'll submit a PR to upstream (you guys), assuming this works. It turns out that it is libgmp that is detecting the underlying processor instruction set at a finer level of detail than typical for gcc builds, and THAT is causing the differences in libsnark.

Contributor

radix42 commented Oct 20, 2016

Testing a further (one character!) patch to depends/Makefile so it respects both HOST and BUILD options passed in by build.sh. As soon as I have results from this gitian build, I'll submit a PR to upstream (you guys), assuming this works. It turns out that it is libgmp that is detecting the underlying processor instruction set at a finer level of detail than typical for gcc builds, and THAT is causing the differences in libsnark.

@radix42

This comment has been minimized.

Show comment
Hide comment
@radix42

radix42 Oct 20, 2016

Contributor

My pull request 1577 should remove the remaining indeterminism from v1.0.0-rc1 builds in libgmp and libsnark

Contributor

radix42 commented Oct 20, 2016

My pull request 1577 should remove the remaining indeterminism from v1.0.0-rc1 builds in libgmp and libsnark

@radix42

This comment has been minimized.

Show comment
Hide comment
@radix42

radix42 Oct 20, 2016

Contributor

I just verified that PR 1577 does indeed remove processor/chipset architecture from the build....of course you'll want to run it through the full test suite.

Contributor

radix42 commented Oct 20, 2016

I just verified that PR 1577 does indeed remove processor/chipset architecture from the build....of course you'll want to run it through the full test suite.

@daira

This comment has been minimized.

Show comment
Hide comment
@daira

daira Oct 20, 2016

Contributor

Assuming fixed by #1577.

Contributor

daira commented Oct 20, 2016

Assuming fixed by #1577.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment