Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clang link time optimization for darwin #19312

Open
wants to merge 9 commits into
base: master
from

Conversation

@dipinhora
Copy link
Contributor

commented Oct 6, 2016

Motivation for this change

Clang link time optimization was not working on OSX.
Resolves #19098.

Trying to follow CONTRIBUTING.md (hopefully I didn't make too many mistakes):
cc @copumpkin (for cctool/darwin stdenv changes)
cc @lovek323 @raskin @viric (for clang 3.6/llvm 3.7/clang 3.7/clang 3.8/clang 3.9 changes)
cc @fpletz @globin (for cc-wrapper changes)

Things done
  • Tested using sandboxing
    (nix.useSandbox on NixOS,
    or option build-use-sandbox in nix.conf
    on non-NixOS)
  • Built on platform(s)
    • NixOS
    • OS X
    • Linux
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

NOTE: Not all tests were successful but I don't believe the issues are caused by any changes in this PR but I am not sure because I didn't do a baseline of the tests without the changes first.

The following is are the results of the testing combinations as represented by checkboxes above:

Technical details

  • System: Mac OSX 10.10.5; Output of uname -a:
Darwin Dipins-MBP.home 14.5.0 Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64 x86_64
  • Nix version: nix-env (Nix) 1.11.4
  • Nixpkgs version: "17.03pre93030.548bcd0"

mac:

NIX_PATH=nixpkgs=/Users/dipinhora/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed
works fine

NIX_PATH=nixpkgs=/Users/dipinhora/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed --option build-use-sandbox true
error:

building path(s) ‘/nix/store/y9vg140f50a2zq4qsdpdavdmlrlhjjpf-clang-wrapper-9.9.9’
error: derivation ‘/nix/store/5jplsyd68iaxw6d1vrvfl0n2xqpvzh8k-clang-wrapper-9.9.9.drv’ specifies a sandbox profile, but this is only allowed when ‘build-use-sandbox’ is ‘relaxed’

NIX_PATH=nixpkgs=/Users/dipinhora/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed --option build-use-sandbox relaxed
error: (known issue: NixOS/nix#759)

building path(s) ‘/nix/store/y9vg140f50a2zq4qsdpdavdmlrlhjjpf-clang-wrapper-9.9.9’
killing process 26411
killing process 26411: Operation not permitted
note: keeping build directory ‘/private/var/folders/tw/867wzcgs2gg1zwkwn0ws8_k80000gn/T/nix-build-clang-wrapper-9.9.9.drv-1’
error: while setting up the build environment: changing into ‘/private/var/folders/tw/867wzcgs2gg1zwkwn0ws8_k80000gn/T/nix-build-clang-wrapper-9.9.9.drv-1’: No such file or directory

NIX_PATH=nixpkgs=/Users/dipinhora/nixpkgs/ nix-shell -p nox --run "nox-review wip" --pure --keep-failed
error:

note: keeping build directory ‘/private/var/folders/tw/867wzcgs2gg1zwkwn0ws8_k80000gn/T/nix-build-python3.5-pytest-2.9.2.drv-0’
error: opening directory ‘/private/var/folders/tw/867wzcgs2gg1zwkwn0ws8_k80000gn/T/nix-build-python3.5-pytest-2.9.2.drv-0/pytest-of-dipinhora/pytest-0/testdir/test_cache_failure_warns0/.cache’: Permission denied
/Users/dipinhora/.nix-profile/bin/nix-shell: failed to build all dependencies

When run a second time:

Traceback (most recent call last):
  File "/nix/store/q9n9wsafqp93rcv31wy1hd0l6dpg45gi-nox-0.0.4/bin/.nox-review-wrapped", line 12, in <module>
    sys.exit(cli())
  File "/nix/store/bkq3sl3kyz1rki2dv6n1wm7xs824x2s2-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/nix/store/bkq3sl3kyz1rki2dv6n1wm7xs824x2s2-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 675, in main
    _verify_python3_env()
  File "/nix/store/bkq3sl3kyz1rki2dv6n1wm7xs824x2s2-python3.5-click-6.6/lib/python3.5/site-packages/click/_unicodefun.py", line 64, in _verify_python3_env
    stderr=subprocess.PIPE).communicate()[0]
  File "/nix/store/wb7zc6hicj05kgncqkrwiv7z9r9mj21d-python3-3.5.2/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/nix/store/wb7zc6hicj05kgncqkrwiv7z9r9mj21d-python3-3.5.2/lib/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'locale'

nixos: (without and with nix.useSandbox = true) (on aws: ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20160907.1 (ami-2ef48339))

NIX_PATH=nixpkgs=/root/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed --max-jobs 8 --cores 8
works fine

NIX_PATH=nixpkgs=/root/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed --max-jobs 8 --cores 8 --option build-use-sandbox true
works fine

NIX_PATH=nixpkgs=/root/nixpkgs/ nix-shell -p nox --run "nox-review wip" --pure --keep-failed --max-jobs 8 --cores 8
error:

Traceback (most recent call last):
  File "/nix/store/2gaxi7l0z59gq7zya7xqi1gkynp9yy3i-nox-0.0.4/bin/.nox-review-wrapped", line 12, in <module>
    sys.exit(cli())
  File "/nix/store/pqb3555sq1anvfp1ax293xk9dw29f2vm-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/nix/store/pqb3555sq1anvfp1ax293xk9dw29f2vm-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 675, in main
    _verify_python3_env()
  File "/nix/store/pqb3555sq1anvfp1ax293xk9dw29f2vm-python3.5-click-6.6/lib/python3.5/site-packages/click/_unicodefun.py", line 119, in _verify_python3_env
    'mitigation steps.' + extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.  Either run this under Python 2 or consult http://click.pocoo.org/python3/ for mitigation steps.

Additional information: on this system no suitable UTF-8
locales were discovered.  This most likely requires resolving
by reconfiguring the locale system.

ubuntu: (on aws: nixos-16.09.666.3738950-x86_64-hvm-ebs (ami-93347684))

NIX_PATH=nixpkgs=/home/ubuntu/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed --max-jobs 8 --cores 8
works fine

NIX_PATH=nixpkgs=/home/ubuntu/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed --option build-use-sandbox true
error:

building path(s) ‘/nix/store/g7a2yvmgs3ffy91dwppsaifrvgwz41xr-bootstrap-gcc-wrapper’
note: keeping build directory ‘/tmp/nix-build-bootstrap-gcc-wrapper.drv-3’
error: cannot change ownership of ‘/nix/store/vjirhdirwq2k84jxsds0473ik8gn1n00-bootstrap-gcc-wrapper.drv.chroot’: Operation not permitted

NIX_PATH=nixpkgs=/home/ubuntu/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed --option build-use-sandbox relaxed
error:

building path(s) ‘/nix/store/g7a2yvmgs3ffy91dwppsaifrvgwz41xr-bootstrap-gcc-wrapper’
note: keeping build directory ‘/tmp/nix-build-bootstrap-gcc-wrapper.drv-2’
error: cannot change ownership of ‘/nix/store/vjirhdirwq2k84jxsds0473ik8gn1n00-bootstrap-gcc-wrapper.drv.chroot’: Operation not permitted

NIX_PATH=nixpkgs=/home/ubuntu/nixpkgs/ nix-shell -p nox --run "nox-review wip" --pure --keep-failed --max-jobs 8 --cores 8
error:

note: keeping build directory ‘/tmp/nix-build-python3.5-pytest-2.9.2.drv-0’
error: opening directory ‘/tmp/nix-build-python3.5-pytest-2.9.2.drv-0/pytest-of-ubuntu/pytest-0/testdir/test_cache_failure_warns0/.cache’: Permission denied
/home/ubuntu/.nix-profile/bin/nix-shell: failed to build all dependencies

When run a second time:

Traceback (most recent call last):
  File "/nix/store/2gaxi7l0z59gq7zya7xqi1gkynp9yy3i-nox-0.0.4/bin/.nox-review-wrapped", line 12, in <module>
    sys.exit(cli())
  File "/nix/store/pqb3555sq1anvfp1ax293xk9dw29f2vm-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/nix/store/pqb3555sq1anvfp1ax293xk9dw29f2vm-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 675, in main
    _verify_python3_env()
  File "/nix/store/pqb3555sq1anvfp1ax293xk9dw29f2vm-python3.5-click-6.6/lib/python3.5/site-packages/click/_unicodefun.py", line 119, in _verify_python3_env
    'mitigation steps.' + extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.  Either run this under Python 2 or consult http://click.pocoo.org/python3/ for mitigation steps.

Additional information: on this system no suitable UTF-8
locales were discovered.  This most likely requires resolving
by reconfiguring the locale system.

@mention-bot

This comment has been minimized.

Copy link

commented Oct 6, 2016

@dipinhora, thanks for your PR! By analyzing the history of the files in this pull request, we identified @vcunat, @copumpkin and @edolstra to be potential reviewers.

@dipinhora

This comment has been minimized.

Copy link
Contributor Author

commented Oct 11, 2016

Please let me know if you have any questions regarding this change or if anything needs to be altered before it can be merged.

@dipinhora

This comment has been minimized.

Copy link
Contributor Author

commented Oct 16, 2016

@copumpkin Seeing as this changes cctools and impacts the darwin stdenv, do you have any questions or concerns regarding any of the changes?

If not, what is the process for getting this merged? Does this need to go into staging first?

@copumpkin

This comment has been minimized.

Copy link
Member

commented Oct 16, 2016

Yeah sorry for not being in touch. My plan was to merge this into staging this weekend with some other changes I was going to make, but staging is currently broken on Darwin so now I'm looking into that. I think it's generally fine though, thanks!

On Oct 16, 2016, at 15:04, Dipin Hora notifications@github.com wrote:

@copumpkin Seeing as this changes cctools and impacts the darwin stdenv, do you have any questions or concerns regarding any of the changes?

If not, what is the process for getting this merged? Does this need to go into staging first?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@copumpkin

This comment has been minimized.

Copy link
Member

commented Oct 17, 2016

So I think that in principle it's fine, but in practice it complicates the stdenv bootstrap a bit. I know you made the changes needed to the bootstrap, but it leads to 3 complete builds of LLVM per stdenv build instead of 1 before this change. I'm hoping to make some changes to make it easier to tie together bootstraps, but for now I'd rather not merge at least the LLVM 3.7 changes because it'll make it a nightmare for me to rebuild the stdenv for the other changes I'm making (since LLVM builds are by far the majority of the stdenv build time this makes the whole thing a lot slower)

@dipinhora

This comment has been minimized.

Copy link
Contributor Author

commented Oct 17, 2016

@copumpkin Thanks for reviewing the changes. Much appreciated.

I had noticed that 3 complete builds of llvm/clang were done but didn't do a before/after comparison so didn't realize it was due to my changes.

The LLVM 3.7 changes at https://github.com/NixOS/nixpkgs/pull/19312/files#diff-cf6aa7f087ebdb6ee5061fdbfad9375aR31 and https://github.com/NixOS/nixpkgs/pull/19312/files#diff-cf6aa7f087ebdb6ee5061fdbfad9375aR41 that set the compatibility version are unfortunately required to make it all work because it's not possible to change the compatibility version that ld looks for using install_name_tool. And without the change ld will only load the libLTO.dylib files with a compatibility version of 3.7. This wouldn't be an issue if the stdenv was using clang 3.8 because llvm 3.8 correctly sets the compatibility version for libLTO.dylib to 1.

One way to avoid the llvm 3.7 changes would be if there was some way to change the compatibility version for libLTO.dylib directly in the ld binary generated (similar to how install_name_tool is used to rename the library that ld references from libLTO.3.7.dylib to libLTO.dylib). Is there some way to accomplish that?

@copumpkin

This comment has been minimized.

Copy link
Member

commented Oct 17, 2016

Interesting. One thing I'd done a few weeks ago was upgrade the darwin stdenv to LLVM 3.8 but some strange things started happening and I didn't have time to figure them out so I undid it and reverted to 3.7.

I don't really know what the best solution is here. My inclination would be to ask you to wait until we get the stdenv to 3.8 and/or the stdenv bootstrapping machinery rework (so we can minimize the LLVM rebuilds) but I don't know how urgently you need this. It doesn't seem like the LLVM stuff changes all that often so it doesn't feel like your PR will bitrot very quickly.

How about in the next few days I start a Hydra build (against one of my custom branches) of a 3.8-based stdenv, and then we can see how that affects this? I'm fearful of putting 3.8 straight into staging because I want to have a full Hydra build to compare to see what we're breaking by upgrading.

@dipinhora

This comment has been minimized.

Copy link
Contributor Author

commented Oct 17, 2016

No rush. I'm currently using a fork of nixpkgs with the changes I need so it's not a blocker for me. I just wanted it in upstream so there's less reason for me to keep a fork and have to build pretty much everything from source instead of being able to rely on the Hydra caches.

I think your plan to try a build of a 3.8 based stdenv and to switch over to it is a good one. Let me know if I can help in converting over to a 3.8 based stdenv.

@vcunat

This comment has been minimized.

Copy link
Member

commented Oct 17, 2016

I'd assume that LTO is also broken on Linux and that it would need a (DY)LD_LIBRARY_PATH change in this PR.

@dipinhora

This comment has been minimized.

Copy link
Contributor Author

commented Oct 17, 2016

@vcunat No, LTO works on Linux. See: PR #19457 for a derivation that uses it.

@FRidh

This comment has been minimized.

Copy link
Member

commented Jan 2, 2017

What is the status of this PR?

@LnL7 LnL7 self-assigned this Jan 7, 2017
@LnL7

This comment has been minimized.

Copy link
Member

commented Jan 7, 2017

I'll let some things build using this and see how it looks.

@LnL7

This comment has been minimized.

Copy link
Member

commented Jan 8, 2017

The stdenvBootstrapTools.x86_64-darwin.test fails with this change.

http://hydra.nixos.org/eval/1320568?compare=trunk

dyld: Library not loaded: @rpath/libLTO.dylib
  Referenced from: /nix/store/c2rh6b2f28d0r4kvsfpc8sbi6b80wgm0-bootstrap-tools/bin/ld
  Reason: image not found
/nix/store/zj9i54l7cyz676y5h4fds10ngwipf3r0-unpack-bootstrap-tools.sh: line 45: 54128 Trace/BPT trap: 5       ld -macosx_version_min 10.7 -arch x86_64 -dylib -o $out/lib/libSystem.B.dylib -compatibility_version 1.0 -current_version 1226.10.1 -reexport_library $out/lib/system/libsystem_c.dylib -reexport_library $out/lib/system/libsystem_kernel.dylib $args
builder for ‘/nix/store/w8g3w6v3m4599xafa0z5vjv54bkiz66g-bootstrap-tools.drv’ failed with exit code 133
@copumpkin

This comment has been minimized.

Copy link
Member

commented Apr 8, 2017

I've now merged an upgrade of the stdenv to 4.0, and plan to get LTO into that soon. Sorry it's taken me so long!

@LnL7 LnL7 referenced this pull request Jul 21, 2017
0 of 8 tasks complete
@dtzWill

This comment has been minimized.

Copy link
Contributor

commented May 18, 2018

Beep boop!

@mmahut

This comment has been minimized.

Copy link
Member

commented Aug 11, 2019

Are there any updates on this pull request, please?

@lilyball

This comment has been minimized.

Copy link
Member

commented Sep 14, 2019

I'm hitting what I think is this problem. I'd love to see an update on this PR.

@nixos-discourse

This comment has been minimized.

Copy link

commented Sep 14, 2019

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-in-distress/3604/7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Darwin
  
Package pain points
You can’t perform that action at this time.