New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

g++: error trying to exec 'objcopy': execvp: No such file or directory #86

Closed
ds-hwang opened this Issue Mar 19, 2014 · 11 comments

Comments

Projects
None yet
8 participants
@ds-hwang

ds-hwang commented Mar 19, 2014

Following option of gcc 4.8 uses objcopy, but icecc-create-env does not handle objcopy
'cflags': ['-gsplit-dwarf'], 'ldflags': ['-Wl,--gdb-index'],

So compile fails with following build error

[60/10095] CXX obj/third_party/angle/src/compiler/preprocessor/preprocessor.MacroExpander.o
FAILED: c++ -MMD -MF obj/third_party/angle/src/compiler/preprocessor/preprocessor.MacroExpander.o.d -DV8_DEPRECATION_WARNINGS -DBLINK_SCALE_FILTERS_AT_RECORD_TIME '-DANGLE_PRELOADED_D3DCOMPILER_MODULE_NAMES={ TEXT("d3dcompiler_46.dll"), TEXT("d3dcompiler_43.dll") }' -D_FILE_OFFSET_BITS=64 -DDISABLE_NACL -DCHROMIUM_BUILD -DCOMPONENT_BUILD -DTOOLKIT_VIEWS=1 -DUI_COMPOSITOR_IMAGE_TRANSPORT -DUSE_AURA=1 -DUSE_CAIRO=1 -DUSE_GLIB=1 -DUSE_DEFAULT_RENDER_THEME=1 -DUSE_LIBJPEG_TURBO=1 -DUSE_X11=1 -DUSE_CLIPBOARD_AURAX11=1 -DENABLE_ONE_CLICK_SIGNIN -DUSE_XI2_MT=2 -DENABLE_REMOTING=1 -DENABLE_WEBRTC=1 -DUSE_PROPRIETARY_CODECS -DENABLE_PEPPER_CDMS -DENABLE_CONFIGURATION_POLICY -DENABLE_INPUT_SPEECH -DENABLE_NOTIFICATIONS -DUSE_UDEV -DDCHECK_ALWAYS_ON=1 -DENABLE_EGLIMAGE=1 -DENABLE_TASK_MANAGER=1 -DENABLE_EXTENSIONS=1 -DENABLE_PLUGIN_INSTALLATION=1 -DENABLE_PLUGINS=1 -DENABLE_SESSION_SERVICE=1 -DENABLE_THEMES=1 -DENABLE_AUTOFILL_DIALOG=1 -DENABLE_BACKGROUND=1 -DENABLE_AUTOMATION=1 -DENABLE_GOOGLE_NOW=1 -DCLD_VERSION=2 -DENABLE_FULL_PRINTING=1 -DENABLE_PRINTING=1 -DENABLE_SPELLCHECK=1 -DENABLE_CAPTIVE_PORTAL_DETECTION=1 -DENABLE_APP_LIST=1 -DENABLE_SETTINGS_APP=1 -DENABLE_MANAGED_USERS=1 -DENABLE_MDNS=1 -DUSE_NSS=1 -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DWTF_USE_DYNAMIC_ANNOTATIONS=1 -D_DEBUG -D_GLIBCXX_DEBUG=1  -fstack-protector --param=ssp-buffer-size=4 -pthread -fno-exceptions -fno-strict-aliasing -Wno-unused-parameter -Wno-missing-field-initializers -fvisibility=hidden -pipe -fPIC -Wno-unused-local-typedefs -Wno-format -Wno-unused-result -m64 -march=x86-64 -O0 -g -funwind-tables -gsplit-dwarf -fno-rtti -fno-threadsafe-statics -fvisibility-inlines-hidden -Wno-deprecated  -c ../../third_party/angle/src/compiler/preprocessor/MacroExpander.cpp -o obj/third_party/angle/src/compiler/preprocessor/preprocessor.MacroExpander.o
g++: error trying to exec 'objcopy': execvp: No such file or directory
ICECC[9635] 12:16:59: Compiled on 10.237.72.78
@doomdavve

This comment has been minimized.

Show comment
Hide comment
@doomdavve

doomdavve Mar 19, 2014

Adding objcopy to the tarball is the easy part. Passing back the .dwo file after split from the nodes is probably more work and would probably require a protocol bump(?).

Another idea is to handle the flag on the client; split the .o file after it has returned from where it's compiled. But that requires a little too much implementation knowledge of gcc for my taste; icecc would need to know exactly how to split the .o file.

doomdavve commented Mar 19, 2014

Adding objcopy to the tarball is the easy part. Passing back the .dwo file after split from the nodes is probably more work and would probably require a protocol bump(?).

Another idea is to handle the flag on the client; split the .o file after it has returned from where it's compiled. But that requires a little too much implementation knowledge of gcc for my taste; icecc would need to know exactly how to split the .o file.

@ds-hwang

This comment has been minimized.

Show comment
Hide comment
@ds-hwang

ds-hwang Mar 19, 2014

aha, it's why I even failed after manually adding objcopy to the tarball.

ds-hwang commented Mar 19, 2014

aha, it's why I even failed after manually adding objcopy to the tarball.

@llunak

This comment has been minimized.

Show comment
Hide comment
@llunak

llunak Mar 19, 2014

Contributor

And obvious "fix" for this is forcing local build in client/arg.cpp if -gsplit-dwarf is found.

From a quick glance, it doesn't look like icecc could easily handle the flag itself, since although objcopy has --extract-dwo and --strip-dwo, presumably gcc ouputs different .o files depending on the switch.

But passing the .dwo file from the remote shouldn't be that much work. Extra flag in CompilerResultMsg (and yes, protocol bump) and using extra FileChunkMsg in daemon/workit.cpp and client/remote.cpp shouldn't be too difficult.

-Wl,--gdb-index are used during linking, so that's of no concern to icecream.

Contributor

llunak commented Mar 19, 2014

And obvious "fix" for this is forcing local build in client/arg.cpp if -gsplit-dwarf is found.

From a quick glance, it doesn't look like icecc could easily handle the flag itself, since although objcopy has --extract-dwo and --strip-dwo, presumably gcc ouputs different .o files depending on the switch.

But passing the .dwo file from the remote shouldn't be that much work. Extra flag in CompilerResultMsg (and yes, protocol bump) and using extra FileChunkMsg in daemon/workit.cpp and client/remote.cpp shouldn't be too difficult.

-Wl,--gdb-index are used during linking, so that's of no concern to icecream.

@doomdavve

This comment has been minimized.

Show comment
Hide comment
@doomdavve

doomdavve commented Mar 19, 2014

FWIW, Ccache gives up (as a first step): https://bugzilla.samba.org/show_bug.cgi?id=10005#c3

@ds-hwang

This comment has been minimized.

Show comment
Hide comment
@ds-hwang

ds-hwang Mar 19, 2014

I filed this bug because chromium began using the noxious options: https://code.google.com/p/chromium/issues/detail?id=352046
Google dev uses their own internal distributed build system, so called gorma: https://groups.google.com/a/chromium.org/forum/#!topic/clang/WR07rBBzGuk
I guess gorma can handle it.

ds-hwang commented Mar 19, 2014

I filed this bug because chromium began using the noxious options: https://code.google.com/p/chromium/issues/detail?id=352046
Google dev uses their own internal distributed build system, so called gorma: https://groups.google.com/a/chromium.org/forum/#!topic/clang/WR07rBBzGuk
I guess gorma can handle it.

@mithro

This comment has been minimized.

Show comment
Hide comment
@mithro

mithro Apr 26, 2014

Note that this option is supported by clang too (it is not a GCC specific thing).

This option gives massive speed increases when linking. On an incremental build of Chrome (as with most large C++ projects) the linking stage is the most substantial part of the build. It saves between ~30% and ~80% of the link time (with most seeing around 50%).

Mozilla are also using this option for similar results. LibreOffice is also playing with enabling it by default.

It is likely that every large C++ project is going to use this option in the next 2-3 years.

Hope that gives you some more info.

mithro commented Apr 26, 2014

Note that this option is supported by clang too (it is not a GCC specific thing).

This option gives massive speed increases when linking. On an incremental build of Chrome (as with most large C++ projects) the linking stage is the most substantial part of the build. It saves between ~30% and ~80% of the link time (with most seeing around 50%).

Mozilla are also using this option for similar results. LibreOffice is also playing with enabling it by default.

It is likely that every large C++ project is going to use this option in the next 2-3 years.

Hope that gives you some more info.

@Tomasu

This comment has been minimized.

Show comment
Hide comment
@Tomasu

Tomasu May 2, 2014

I would be interested in seeing support for this. I have a project that uses clang and llvm libs, and the link time when in debug mode are quite long.

Tomasu commented May 2, 2014

I would be interested in seeing support for this. I have a project that uses clang and llvm libs, and the link time when in debug mode are quite long.

jyasskin pushed a commit to jyasskin/chromium that referenced this issue May 5, 2014

mostynb@opera.com
add a linux_use_debug_fission gyp variable for -gsplit-dwarf
Since -gsplit-dwarf breaks some extremely useful build tools (eg ccache and icecc), it deserves its own gyp variable.  This setting replaces the previous hack, setting binutils_version=0.

Background:
ccache: https://bugzilla.samba.org/show_bug.cgi?id=10005
icecc: icecc/icecream#86

BUG=352046

Review URL: https://codereview.chromium.org/226613005

git-svn-id: svn://svn.chromium.org/chrome/trunk/src@267381 0039d316-1c4b-4281-b951-d872f2087c98
@coolo

This comment has been minimized.

Show comment
Hide comment
@coolo

coolo Oct 24, 2014

Contributor

I close this one as the bug is fixed by forcing local compile. There doesn't seem to be enough interest to really implement the feature

Contributor

coolo commented Oct 24, 2014

I close this one as the bug is fixed by forcing local compile. There doesn't seem to be enough interest to really implement the feature

@coolo coolo closed this Oct 24, 2014

@ostapru

This comment has been minimized.

Show comment
Hide comment
@ostapru

ostapru Oct 29, 2014

What it means "doesn't seem to be enough interest"?
What is the threshold to be "enough interest"?

In general, for me it is very critical issue, because debug fission is not only improve linking speed, but also improves gdb speed during debugging.
In general, I stopped using icecc for debug builds now, because it only improves full rebuilds, but incremental builds and debugging are slower without debug fission.

ostapru commented Oct 29, 2014

What it means "doesn't seem to be enough interest"?
What is the threshold to be "enough interest"?

In general, for me it is very critical issue, because debug fission is not only improve linking speed, but also improves gdb speed during debugging.
In general, I stopped using icecc for debug builds now, because it only improves full rebuilds, but incremental builds and debugging are slower without debug fission.

@coolo

This comment has been minimized.

Show comment
Hide comment
@coolo

coolo Oct 29, 2014

Contributor

the threshold is the existance of a patch

Contributor

coolo commented Oct 29, 2014

the threshold is the existance of a patch

@clopez

This comment has been minimized.

Show comment
Hide comment
@clopez

clopez Sep 16, 2015

Contributor

For the record.
I'm succesfully using ccache+icecc+[clang|gcc] with -gsplit-dwarf.

To make it work you have to take into account the following:

  • If you use ccache in the mix, ensure your ccache version is >= 3.2.3 (3.2.2 Don't has support for -gsplit-dwarf)
  • You have to create a tarball for your toolchain like this
    • For Clang

      /usr/lib/icecc/icecc-create-env --clang /usr/bin/clang /usr/lib/icecc/compilerwrapper --addfile /usr/bin/objcopy
      
    • For GCC

      /usr/lib/icecc/icecc-create-env --gcc /usr/bin/gcc /usr/bin/g++ --addfile /usr/bin/objcopy
      
  • After that, export the env variable ICECC_VERSION to the full path containing that file

Take into account also the following:

  • I tried this with Clang 3.6.1 and GCC 4.9.2. If you are using an older version of either Clang or GCC and this don't works for you, try upgrading it.
  • There is an issue, that would cause the buildslaves to fill the disk very quickly. So take care of this. I have reported it on #140
Contributor

clopez commented Sep 16, 2015

For the record.
I'm succesfully using ccache+icecc+[clang|gcc] with -gsplit-dwarf.

To make it work you have to take into account the following:

  • If you use ccache in the mix, ensure your ccache version is >= 3.2.3 (3.2.2 Don't has support for -gsplit-dwarf)
  • You have to create a tarball for your toolchain like this
    • For Clang

      /usr/lib/icecc/icecc-create-env --clang /usr/bin/clang /usr/lib/icecc/compilerwrapper --addfile /usr/bin/objcopy
      
    • For GCC

      /usr/lib/icecc/icecc-create-env --gcc /usr/bin/gcc /usr/bin/g++ --addfile /usr/bin/objcopy
      
  • After that, export the env variable ICECC_VERSION to the full path containing that file

Take into account also the following:

  • I tried this with Clang 3.6.1 and GCC 4.9.2. If you are using an older version of either Clang or GCC and this don't works for you, try upgrading it.
  • There is an issue, that would cause the buildslaves to fill the disk very quickly. So take care of this. I have reported it on #140

qtprojectorg pushed a commit to qt-creator/qt-creator that referenced this issue Jan 23, 2017

hjk
Blacklist -gsplit-dwarf produced by llvm-config
This requires objcopy which might not (easily) be available when
using icecc: icecc/icecream#86

Change-Id: I6e152190949d945c343837ba433daf75b0ba7ddd
Reviewed-by: Nikolai Kosjar <nikolai.kosjar@qt.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment