release: use static libstdc++ and disable reduced exports by default

[theuni]

Since the release of 0.10.0, we've seen several reports of failed sanity tests due to failing to catch basic exceptions. At the moment, Debian Wheezy fails to run the official Linux binary, and FreeBSD fails to run when compiled natively. I assume that there are several other broken setups as well.

From what I've seen, the brokenness is always due to type_info visibility. Building with -fvisibility=hidden can cause problems with old/busted stdlibs. See here for some background.

It's nearly impossible to tell (without blacklisting std libs like we already do with boost) if libstdc++ is going to play nice with --enable-reduce-exports. Rather than trying, just disable it by default. Packagers can re-enable if it's known to be safe.

For Linux, when compiling against a new libstdc++ but using an older one at runtime, exceptions can again be missed. Rather than take any chances, use -static-libstdc++. This means we don't have to worry about trying to maintain a compat wrapper.

Binaries become slightly bigger (about 1M each), but not enough to worry about imo. The final compressed sizes:

before:
17416   built/linux/bitcoin-0.10.0-linux32.tar.gz
17196   built/linux/bitcoin-0.10.0-linux64.tar.gz

after:
18920   build/out/bitcoin-0.10.99-linux32.tar.gz
18804   build/out/bitcoin-0.10.99-linux64.tar.gz

Debian Wheezy is verified fixed. @koobs has verified that building without reduced exports fixes freebsd.

[theuni]

ping @gmaxwell, your thoughts would be appreciated since we discussed this a few nights ago.

[gmaxwell]

Sad to hear that visibility control is being problematic, especially as we move towards making a library out of things. In the long term I don't think we can have the library that way (or callers will use random internal symbols). In the short term sure.

The size increase is smaller than I expected; 8% is worth a reduction in consensus risk and getting binaties working on debian wheezy esp considering how unstable in the fine details stdc++ has been historically. I'm not aware of any problems with static stdc++ of the sort that libc, and many commercial applications do this already. So indeed, it sounds good to me.

[koobs]

@theuni While we're here, --disable-reduced-exports in the error messaging needs to be changed to --disable-reduce-exports

[theuni]

@koobs It's fixed, see 3448b13.

@gmaxwell As far as I'm aware, all trouble is caused by old (gcc 4.2 era) libstdc++. So eventually this won't be much of a problem (or it will be replaced with newer problems...).

But more to your point, I suppose I've explained the changes here poorly. For our official releases, we still build with hidden symbols as before, we just also link libstdc++ statically. The change here in that regard is that --disable-reduced-exports becomes the default in our configure... builders can re-enable if they'd like (as we do). So it should be effectively a no-op as far as official release symbol visibility goes.

[gmaxwell]

@theuni I followed wrt visibility and releases. I /think/ virtually no one using the software as a library will be using our release binaries, so we can expect the excessive exports being abused. I think thats acceptable for now as none of the library interface stuff is stable to begin with.

[koobs]

@theuni Just for additional reference here, your statement about libstdc++ being the issue is confirmed and evidenced in the fact that whether using cc (gcc 4.2.1) or clang (3.4.1) on FreeBSD 9-STABLE, libstdc++ is linked.

clang can use both GNU libstdc++ and libc++, depending on the -stdlib= option. On 9.x it uses the former by default, on 10.x and later the latter.

[laanwj]

Good catch. Again shows that shipping portable binaries for Linux is all but impossible.
Maybe something like xdg-app sandboxing can change this in the near future.

One question: Won't statically linking libstdc++ in the gitian build give problems with Qt (as we don't like that statically, and thus the dynamic qt will imports its own libstdc++)?

[theuni]

@laanwj I'm really unsure. As an initial data point, it appears to run fine on Debian Wheezy (where it was broken before).

Thinking through it a bit...
bitcoin-qt's own throw/catch usage should be fine because the exception types are defined internally. Qt's throw/catch usage should be fine because their types are dynamically loaded, and we don't export them.

I think the worrisome case would be if qt throws us a std::exception. In that case, I'm really not sure what definition is used. Though from a quick grep, it looks like that never happens.

That's all conjecture though. I'd like to have hard evidence. Beyond expanding our sanity checks to other std exceptions, can you think of other tests we could add here?

[laanwj]

@theuni Maybe I'm misunderstanding something. I'm not only worried about exceptions, but as I see it this will result in Qt and bitcoin-qt using different versions of libstdc++. The two versions of libstdc++ may have slightly different interpretations of types. When stdc++ types are passed across the dynamic library barrier (for example for QString::fromStdStding) this could cause potential issues. Another potential cause of issues are global data structures in libstdc++.

I know this is a common issue on Windows, at least. Maybe there is some magic workaround, but it seems brittle to me. A way to invite unknown unknowns. Hence I'm not so much convinced by 'it happens to work on X'.

The recommendation when statically linking libstdc++ statically is to also link all c++ libraries statically. See e.g. http://www.trilithium.com/johan/2005/06/static-libstdc . Dynamic linking of C libraries is fine.

For Linux, when compiling against a new libstdc++ but using an older one at runtime

• The caveman solution that comes to mind is: build against an old libstdc++.
• Or should we bite the bullet and link Qt statically. More and more that sounds like the only robust solution (remember the OpenSSL interop issues). Static qt5 sucks on Ubuntu, but that is cosmetic and preferable to these kinds of issues.

[gmaxwell]

Well for example, GCC 5's libstdc++ includes an ABI incompatible string type. ... but there I expect bitcoin core won't work at all on those systems unless all C++ parts are statically linked.

[laanwj]

@gmaxwell Right. Running the binary on such a system would only work if Qt is statically linked as well. Another point in favor of that.

[theuni]

@laanwj I'm all for static qt if you're up for it. I'll run a quick gitian test with the above changes + static qt to see if there are any surprises.

[theuni]

Changed to static qt. Release sizes roughly doubled :(

For testing:
https://bitcoincore.org/cfields/static-qt/bitcoin-0.10.99-linux32.tar.gz
https://bitcoincore.org/cfields/static-qt/bitcoin-0.10.99-linux64.tar.gz

Works fine on Debian Squeeze 32.

binary depends:

 0x0000000000000001 (NEEDED)             Shared library: [libX11-xcb.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libX11.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libxcb.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libfontconfig.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libfreetype.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libanl.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]

I'm not sure whether we should build qt using -no-xcb-xlib or not. That would drop the libX11-xcb.so.1 and libX11.so.6 deps, but I'm not clear on the implications.

[sipa]

No opinion.

[laanwj]

That would drop the libX11-xcb.so.1 and libX11.so.6 deps, but I'm not clear on the implications.

Those are extremely common libraries, and also purely C not C++, so I have no problem with having them as dependency for the GUI.