Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build/release: create new release package for reproducible builds #3545

Merged
merged 1 commit into from Oct 4, 2019

Conversation

@Roasbeef
Copy link
Member

Roasbeef commented Sep 26, 2019

In this commit, we create a new build/release package which houses the
container and scripts that we need to do deterministic builds across
Windows, Linux, and MacOS. As a result, all builds of lnd will now
take place from within this container environment. The container will
mount to the local lnd directory, resulting in all the build artifacts
being present in the host's cwd.

With this new system, it's now possible for all lnd developers, as
well as users to verify the posted build binaries. This wasn't possible
in prior release as only in Go 1.13 did deterministic builds becomes
easier/possible. See the new README.md for further details.

@Roasbeef Roasbeef force-pushed the Roasbeef:reproducible-builds branch 5 times, most recently from ab9f15b to 33ebcb0 Sep 27, 2019
Copy link
Collaborator

guggero left a comment

Nice to see we finally can provide reproducible builds!
It didn't work for me though (on Ubuntu). Had to make some changes (see comments) then it worked like a charm.

build/release/README.md Outdated Show resolved Hide resolved
build/release/README.md Outdated Show resolved Hide resolved
build/release/Dockerfile Outdated Show resolved Hide resolved
@Roasbeef

This comment has been minimized.

Copy link
Member Author

Roasbeef commented Sep 27, 2019

@Roasbeef Roasbeef force-pushed the Roasbeef:reproducible-builds branch from 33ebcb0 to acda68e Oct 1, 2019
@Roasbeef

This comment has been minimized.

Copy link
Member Author

Roasbeef commented Oct 1, 2019

Alright, push up a new version, that should be working now!

@Roasbeef

This comment has been minimized.

Copy link
Member Author

Roasbeef commented Oct 1, 2019

My build for linux-amd64 on this commit: 59162ece3287061bcdc2a81ce637b9a3f4bd9e986440325b9a4782c2adc48df7

Built using:

LNDBUILDSYS=linux-amd64 ./build/release/release.sh v0.8.0-rc2
@Roasbeef

This comment has been minimized.

Copy link
Member Author

Roasbeef commented Oct 1, 2019

It also should be the case that on linux the directory no longer matters, so it can be built from anywhere.

@Roasbeef

This comment has been minimized.

Copy link
Member Author

Roasbeef commented Oct 1, 2019

Confirmed I can do the same release on MacOS and it matches that lnd binary hash above.

@wpaulino

This comment has been minimized.

Copy link
Collaborator

wpaulino commented Oct 1, 2019

This will require a clean git working tree and go1.13.1.

build/release/README.md Outdated Show resolved Hide resolved
build/release/README.md Outdated Show resolved Hide resolved
build/release/README.md Outdated Show resolved Hide resolved
build/release/README.md Outdated Show resolved Hide resolved
build/release/README.md Outdated Show resolved Hide resolved
build/release/README.md Outdated Show resolved Hide resolved
build/release/README.md Outdated Show resolved Hide resolved
build/release/README.md Outdated Show resolved Hide resolved
build/release/README.md Outdated Show resolved Hide resolved
build/release/README.md Outdated Show resolved Hide resolved
build/release/release.sh Show resolved Hide resolved
@guggero

This comment has been minimized.

Copy link
Collaborator

guggero commented Oct 2, 2019

Can confirm that with linux and go 1.13.1 I can build inside my home directory and the hash matches.

@Roasbeef Roasbeef force-pushed the Roasbeef:reproducible-builds branch from acda68e to 8cc2622 Oct 3, 2019
@Roasbeef Roasbeef requested review from wpaulino and cfromknecht Oct 3, 2019
@guggero

This comment has been minimized.

Copy link
Collaborator

guggero commented Oct 3, 2019

Did some tests on Windows 10 today:

  • git bash (cygwin), from source directly: did NOT create correct hash on binaries
  • WSL (Ubuntu on Windows), from source directly: ✔️
  • docker on Windows:
    • source must be checked out with git config --global core.autocrlf false, otherwise release.sh won't start in the container
    • had some problems with the vendor tar, got error message tar: file changed as we read it. probably related to the way docker uses Windows file system and maybe even virus scanner.
    • commented out everything but build instructions and finally got matching hashes for the binaries ✔️

Summary: We should recommend Windows users to go for the WSL option (from source, without docker) if they want to do reproducible builds. Then we could get rid of the container altogether.

@wpaulino wpaulino added the v0.8.0 label Oct 3, 2019
@wpaulino wpaulino added this to the 0.8.0 milestone Oct 3, 2019
Copy link
Collaborator

wpaulino left a comment

Thanks for those tests @guggero! The container is pretty small so we could leave it for any Git Bash users on Windows, though I'm not sure how many of those we have. I'm fine with removing it as well.

build/release/Dockerfile Outdated Show resolved Hide resolved
build/release/README.md Outdated Show resolved Hide resolved
build/release/README.md Outdated Show resolved Hide resolved
build/release/README.md Outdated Show resolved Hide resolved
build/release/README.md Outdated Show resolved Hide resolved
@Roasbeef Roasbeef force-pushed the Roasbeef:reproducible-builds branch from 8cc2622 to c549409 Oct 3, 2019
@Roasbeef

This comment has been minimized.

Copy link
Member Author

Roasbeef commented Oct 3, 2019

Container removed, instructions updated.

@Roasbeef Roasbeef requested review from wpaulino and guggero Oct 3, 2019
build/release/README.md Outdated Show resolved Hide resolved
build/release/README.md Outdated Show resolved Hide resolved
In this commit, we create a new `build/release` package which houses the
build instructions and scripts that we need to do deterministic builds
across Windows, Linux, and MacOS.

With this new system, it's now possible for all `lnd` developers, as
well as users to verify the posted build binaries. This wasn't possible
in prior release as only in Go 1.13 did deterministic builds becomes
easier/possible. See the new `README.md` for further details.
@Roasbeef Roasbeef force-pushed the Roasbeef:reproducible-builds branch from c549409 to 0e41f07 Oct 4, 2019
@Roasbeef Roasbeef merged commit 46cecb2 into lightningnetwork:master Oct 4, 2019
1 check was pending
1 check was pending
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
binaries are now reproducible, allowing developers to build the binary on
distinct machines, and end up with a byte-for-byte identical binary. However,
this wasn't _fully_ solved in Go 1.13, as the build system still includes the
directory the binary is built into the binary itself. As a result, our scripts

This comment has been minimized.

Copy link
@cfromknecht

cfromknecht Oct 4, 2019

Collaborator

is this last part necessary if we just keep the -buildid= flag going forward to allow any go1.13.x version to be used?

This comment has been minimized.

Copy link
@Roasbeef

Roasbeef Oct 4, 2019

Author Member

Well we'll only do our "official" builds w/ the latest one at the time likely, so after Go 1.13.2 re can remove the buildid stuff, but they'll still need to use that latest version.

With Go 1.13, it's now possible for third parties to verify a release binary.
Before this release, one had to trust that release manager to build the proper
binary. With this new system, third parties can now _independently_ run the
release process, and verify that all the hashes in the final `manifest.txt`
match up.
Comment on lines +30 to +34

This comment has been minimized.

Copy link
@cfromknecht

cfromknecht Oct 4, 2019

Collaborator

should we also note that verification must be done using the same go version?

This comment has been minimized.

Copy link
@Roasbeef

Roasbeef Oct 4, 2019

Author Member

Was thinking to leave it to the release notes to dictate which version is used so we don't need to update this README each time a new version is released.

This comment has been minimized.

Copy link
@cfromknecht

cfromknecht Oct 4, 2019

Collaborator

gotcha, yeah i was thinking here we'd just say something like "when verifying, you must use the same version of go as the releaser. check the release notes for the exact version of go to use for a particular set of binaries."

@molxyz

This comment has been minimized.

Copy link

molxyz commented Oct 5, 2019

@guggero
I'm not sure if WSL can do a good job handling lnd, but WSL2 might be able to since it uses a real Linux kernel (https://devblogs.microsoft.com/commandline/announcing-wsl-2/).

But for those who want to run this build on native Windows command line it's also possible by installing either Cygwin or "Git for Windows" which gives you Linux commands that can run on windows. From what I understand it installs the command sh.exe which makes it possible for windows to build linux scripts . After the install, to make these commands run anywhere you just need to set paths for them in System Environment Variables, eg. C:\Program Files\Git\usr\bin

You can use either Windows PowerShell or Command Prompt to build. This is how I did on Command Prompt:

cd %GOPATH%\src\github.com\lightningnetwork
git clone https://github.com/lightningnetwork/lnd
cd lnd
set LNDBUILDSYS=windows-amd64
build\release\release.sh v0.8.0-rc2

The build creates lnd.exe and lncli.exe in directory %GOPATH%\src\github.com\lightningnetwork\lnd\lnd-v0.8.0-rc2\lnd-windows-amd64-v0.8.0-rc2
Then I copy them over to the other directory cp lncli.exe lnd.exe %GOPATH%\bin (Not sure if there's a better way like "make install" or something?)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.