Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gitian: add FreeBSD support #6118

Merged
merged 10 commits into from
Nov 18, 2019
Merged

gitian: add FreeBSD support #6118

merged 10 commits into from
Nov 18, 2019

Conversation

hyc
Copy link
Collaborator

@hyc hyc commented Nov 11, 2019

No description provided.

@hyc
Copy link
Collaborator Author

hyc commented Nov 12, 2019

@TheCharlatan ping

@hyc hyc force-pushed the giti2 branch 3 times, most recently from 7d2a64b to 19a6a7b Compare November 13, 2019 16:37
@hyc
Copy link
Collaborator Author

hyc commented Nov 13, 2019

The point of this last round of patches was to make the dependencies reproducible as well. Previously they weren't identical because their tarball timestamps changed with the monero revision. Now they're constant, so they should have identical hashes regardless of future monero releases.

@hyc hyc force-pushed the giti2 branch 2 times, most recently from 72b561f to 2743f0b Compare November 14, 2019 01:34
@hyc hyc force-pushed the giti2 branch 5 times, most recently from 837fb8c to 18a63ec Compare November 15, 2019 22:00
@TheCharlatan
Copy link
Contributor

TheCharlatan commented Nov 16, 2019

Despite your boost archiver patch, I still get these errors when I build locally: https://travis-ci.org/TheCharlatan/monero/jobs/612656939?utm_medium=notification&utm_source=github_status .
EDIT: Just saw, that you pushed further fixes, will try again.
EDIT: Still getting the same errors.

@@ -0,0 +1,17 @@
freebsd_CC=clang-8
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this only work for clang-8? Would be nice to get the requirement down to a more generic clang version. It also requires the system to alias its clang installation to clang-8, so probably it should just work clang instead.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-8 is the same version that was actually shipped on FreeBSD 11, which is why I set that. Ubuntu offers clang 7, 8, and 9 and I didn't want to leave it ambiguous.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd leave it generic; it at least compiles fine on my clang 7 system.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that C++ library ABIs are compiler version-dependent, I thought it was safest to use the same version FreeBSD uses. Did you run your binaries? I was able to run mine on a FreeBSD 11 VM.

@hyc
Copy link
Collaborator Author

hyc commented Nov 16, 2019

I was focused on getting the Android build dependencies 100% reproducible (which they are now, at the moment) and I might have broken something else. The freeBSD build was already 100% before my recent changes. I'm re-running all the builds now to see what's broken.

@hyc hyc changed the title gitian: add FreeBSD support [WIP] gitian: add FreeBSD support Nov 16, 2019
@hyc hyc force-pushed the giti2 branch 3 times, most recently from 8d36156 to 74db688 Compare November 17, 2019 06:50
@hyc
Copy link
Collaborator Author

hyc commented Nov 17, 2019

The android, freeBSD, and windows builds are now 100% identical. In the linux builds, the order of objects in libunwind.a still doesn't match; I've added an explicit sort command to libtool to try to fix that. A few of the native_* packages on Mac still don't match.

@hyc hyc force-pushed the giti2 branch 3 times, most recently from d15edaf to ed8e2c3 Compare November 17, 2019 08:20
@hyc
Copy link
Collaborator Author

hyc commented Nov 17, 2019

Linux dependencies are now 100% identical. Just have to look into these Mac discrepancies.

@hyc hyc force-pushed the giti2 branch 2 times, most recently from 569248a to 940d927 Compare November 17, 2019 13:35
@hyc
Copy link
Collaborator Author

hyc commented Nov 17, 2019

Fixed all of the Mac differences but one. There's a known bug in the old version of clang that the SDK uses. https://reviews.llvm.org/D50559 This bug makes the output of the Objective-C compiler completely non-deterministic. As a result, the libobjc2/arc.m file in the native_cctools package gives a different resulting .o file every time it's compiled. I don't see any solution to this with the version of clang in use.

Supposedly turning off ASLR will also prevent this bug from happening, but I couldn't find any way to do so in the container. Trying to change /proc/sys/kernel/randomize_va_space fails with EROFS and trying to use setarch -R fails with EPERM.

Note that since this only affects one of the tools in the Mac toolchain, and apparently is only used when compiling objective C, it has never had any impact on the reproducibility of the Monero code. So there's nothing wrong with our previous Mac builds.

I just wanted all the dependencies to be 100% identical so that in the future, if we get a build discrepancy, it's easier to narrow down which component diverged. (Instead of like the last time I had a problem, and didn't know if the problem was in my dependencies or in my Monero tree.) We'll just have to know to ignore differences in the Mac native_cctools package.

Needed to invoke deterministic mode on some archivers
streamline, remove obsolete wrappers
get rid of sodium-darwin.mk special case, unify back with sodium.mk
Sort objects being archived
@hyc hyc changed the title [WIP] gitian: add FreeBSD support gitian: add FreeBSD support Nov 17, 2019
@hyc
Copy link
Collaborator Author

hyc commented Nov 18, 2019

When all else fails, delete the offending binary... Mac dependencies are now 100% identical.

By omitting the otool binary which is built non-deterministically.
We don't use it anyway.
luigi1111 added a commit that referenced this pull request Nov 18, 2019
@luigi1111 luigi1111 merged commit abeed9a into monero-project:master Nov 18, 2019
@hyc hyc deleted the giti2 branch November 19, 2019 16:39
@@ -4,6 +4,7 @@ $(package)_download_path=https://dl.bintray.com/boostorg/release/1.64.0/source/
$(package)_file_name=$(package)_$($(package)_version).tar.bz2
$(package)_sha256_hash=7bcc5caace97baa948931d712ea5f37038dbb1c5d89b43ad4def4ed7cb683332
$(package)_dependencies=libiconv
$(package)_patches=fix_aroptions.patch
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to note - this patch has now been merged into upstream boost. boostorg/build#507

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants