incompatibility w/gcc re: ODR, template instantiation, optimization across translation units

[llvmbot]

Bugzilla Link	27853
Resolution	INVALID
Resolved on	May 26, 2016 18:02
Version	trunk
OS	Linux
Attachments	demonstration of described behavior
Reporter	LLVM Bugzilla Contributor
CC	@dwblaikie,@DougGregor,@zygoloid,@rnk

Extended Description

We've noticed a major incompatability between clang & gcc in the way that template classes are instantiated when optimization is in effect across multiple translation units.

There's not much to be found in a Google search, but this post (https://whatofhow.wordpress.com/2015/03/17/odr-rtti-dso/) seems to suggest that this incompatibility is intentional.

This recently bit us with code similar to attached, which was attempting to define a single using a local static object. The code works as expected with g++ (4.3.3, 4.8.2, 5.3.0) at all optimization levels. Also works as expected with clang at -O0, but at -O3 the instance() method (and associated local static) appear to be inlined, which appears to violate ODR(?). The results are consistent with clang 3.7.0, trunk(r264914) and trunk(r269323).

The results of running the attached code with different compilers and optimization levels:

$ CXX=g++ ./test.sh -O0
test1.so:0000000000000717 W Singleton::instance()
test2.so:0000000000000717 W Singleton::instance()
test1.so:0000000000200a80 u Singleton::instance()::obj
test2.so:0000000000200a80 u Singleton::instance()::obj
$ CXX=g++ ./test.sh -O1
test1.so:0000000000200910 u Singleton::instance()::obj
test2.so:0000000000200910 u Singleton::instance()::obj
$ CXX=g++ ./test.sh -O2
test1.so:0000000000200910 u Singleton::instance()::obj
test2.so:0000000000200910 u Singleton::instance()::obj
$ CXX=g++ ./test.sh -O3
test1.so:0000000000200910 u Singleton::instance()::obj
test2.so:0000000000200910 u Singleton::instance()::obj
$ CXX=clang++ ./test.sh -O0
test1.so:0000000000000770 W Singleton::instance()
test2.so:0000000000000770 W Singleton::instance()
test1.so:0000000000200af0 V Singleton::instance()::obj
test2.so:0000000000200af0 V Singleton::instance()::obj
$ CXX=clang++ ./test.sh -O1
test1.so:0000000000000740 W Singleton::instance()
test2.so:0000000000000740 W Singleton::instance()
test1.so:0000000000200a90 V Singleton::instance()::obj
test2.so:0000000000200a90 V Singleton::instance()::obj
$ CXX=clang++ ./test.sh -O2
$ CXX=clang++ ./test.sh -O3
$

So, I have the following questions:

• Is this behavior intended?
• Is this behavior correct?

Any references to rationale for this discrepancy would be much appreciated.

Thanks!

[dwblaikie]

I'm probably missing something (perhaps this is obvious to others) but could you explain which part of this behavior you believe violates the C++ standard?

It seems to me that since all definitions of Singleton must be identical, and all definitions of Test must be identical, then inlining it all away to just the constants 84 and 43 seems correct...

[llvmbot]

To be clear, the behavior that we observe with clang is that each translation unit gets its own instantiation of the template class, instance() method and local static obj.

In other words, taking the address of instance() gives a different result depending on which translation unit is doing it.

In the sample code this makes no difference, but in our application this matters a lot -- we rely on the instance() method returning a reference to the one-and-only local static obj that should exist in the executable.

With gcc this is the behavior we get. With clang with optimization it is not -- each translation unit instantiates its own copy.

I don't know if this behavior violates the ODR (hence the ? in original question), but it sure is different from the way gcc works, and what a reasonable person would expect.

Does that help clarify things?

[llvmbot]

To expand a bit:

• In our app the Test class is (very) heavy-weight, and maintains network connections, collections of other objects, etc.

• One instance of the class gets fully initialized with the above resources, and so is actually able to do work.

• Any other instances do not -- so the instances are NOT interchangeable.

• That is why we depend on there being one-and-only-one instance.

Again, hth.

[rnk]

The difference between "V" and "u" in the nm output is interesting. The nm man page says:

       "u" The symbol is a unique global symbol.  This is a GNU extension to the standard set of ELF symbol bindings.  For such a symbol the dynamic linker will make sure that in the entire process there is just one symbol with this name and type in use.

       "V"
       "v" The symbol is a weak object.  When a weak defined symbol is linked with a normal defined symbol, the normal defined symbol is used with no error.  When a weak undefined symbol is linked and the symbol is not defined, the value of the weak symbol becomes zero with no error.  On some systems, uppercase indicates that a default value has been specified.

I thought "V" also achieved the same effect as "u", i.e. if you have multiple weak symbols the loader picks one as the winner.

[llvmbot]

More telling, perhaps, is that with clang at -O2 and above there are no symbols at all in the shared lib. It appears that clang has completely inlined not just the static method, but the local static declared in the method.

[rnk]

More telling, perhaps, is that with clang at -O2 and above there are no
symbols at all in the shared lib. It appears that clang has completely
inlined not just the static method, but the local static declared in the
method.

Well, your test code doesn't use the singleton for anything, so we probably discarded it as unreferenced. That should be valid.

[rnk]

test case showing singleton pattern works

[rnk]

I made a more complete test case to show that in normal linking scenarios, your singleton pattern behaves as it should:

$ cat t.cpp
void t1();
void t2();
int main() {
t1();
t2();
}

$ cat t1.cpp
#include <stdio.h>
inline int *singleton() {
static int o;
return &o;
}
void t1() { printf("%p\n", singleton()); }

$ cat t2.cpp
#include <stdio.h>
inline int *singleton() {
static int o;
return &o;
}
void t2() { printf("%p\n", singleton()); }

$ cat doit.sh
clang++ -g -O2 t1.cpp -fPIC -shared -o libt1.so
clang++ -g -O2 t2.cpp -fPIC -shared -o libt2.so
clang++ -g -O2 t.cpp -o t.exe -L. -Wl,-rpath=$ORIGIN -lt1 -lt2
nm libt1.so | grep singleton
nm libt2.so | grep singleton
./t.exe

$ ./doit.sh
000000000020103c V _ZZ9singletonvE1o
000000000020103c V _ZZ9singletonvE1o
0x7f0d9d8ab03c
0x7f0d9d8ab03c

We have two symbols in each shared library, and the loader correctly coalesces them to a single instance.

[llvmbot]

How does the compiler know when building a shared library whether a particular symbol is ever used or not?

The example does not contain an executable, and that was deliberate.

[rnk]

How does the compiler know when building a shared library whether a
particular symbol is ever used or not?

The symbol in question must be emitted in every TU that uses it, so we can safely discard it, knowing that if someone else were to use it, they would be required to emit their own copy.

[llvmbot]

OK, I'm going to need to come up with a better demonstrator then. I was hoping to keep it simple, but that is not to be :-(

In the meantime, though:

• The behavior I am inquiring about is definitely happening. We've noticed different behavior where each TU gets its own copy of the singleton, and then gone back and instrumented the code to log the address of the reference returned by instance() and they are definitely different.

• The actual sequence of symbol resolution is a bit more complicated than presented here. Our application consists of a bunch of shared libraries which are loaded into a Java VM using JNI. These shared libraries then load additional shared libraries ("plugins") using dlopen. Our code uses RTLD_GLOBAL to load libraries, and we think the JVM uses RTLD_LAZY.

• From the blog post referenced earlier(https://whatofhow.wordpress.com/2015/03/17/odr-rtti-dso/), it was starting to sound like the behavior of clang was intended, and in fact technically correct. But since we've been unable to find any supporting documentation, at this point we're mostly just trying to figure out whether the observed behavior is what we should expect.

So, like I said I'll try to put together a better demonstrator and will repost here when I have one. This may take a couple of days, given all the other stuff on my plate.

So, thanks for all the help up to this point, and I hope to be posting back here soon.

Regards,

Bill

[llvmbot]

How does the compiler know when building a shared library whether a
particular symbol is ever used or not?

The symbol in question must be emitted in every TU that uses it, so we can
safely discard it, knowing that if someone else were to use it, they would
be required to emit their own copy.

In the following (amended) example, we clearly see doTest1() and doTest2() in each .so, but no symbol for either the method or local static.

[/home/bt/work/singleton] CXX=clang++ bash -v ./test.sh -O3
#!/bin/bash

${CXX} $* -g -o test1.so -shared -lc -fPIC -lrt test1.cpp
${CXX} $* -g -o test2.so -shared -lc -fPIC -lrt test2.cpp

nm -A *.so |
sort -k 3 -k 1 -s |
c++filt |
egrep 'instance|doTest'
test1.so:0000000000000630 T doTest1()
test2.so:0000000000000630 T doTest2()
[/home/bt/work/singleton]

[llvmbot]

How does the compiler know when building a shared library whether a
particular symbol is ever used or not?

The symbol in question must be emitted in every TU that uses it, so we can
safely discard it, knowing that if someone else were to use it, they would
be required to emit their own copy.

In the following (amended) example, we clearly see doTest1() and doTest2()
in each .so, but no symbol for either the method or local static.

[/home/bt/work/singleton] CXX=clang++ bash -v ./test.sh -O3
#!/bin/bash

${CXX} $* -g -o test1.so -shared -lc -fPIC -lrt test1.cpp
${CXX} $* -g -o test2.so -shared -lc -fPIC -lrt test2.cpp

nm -A *.so |
sort -k 3 -k 1 -s |
c++filt |
egrep 'instance|doTest'
test1.so:0000000000000630 T doTest1()
test2.so:0000000000000630 T doTest2()
[/home/bt/work/singleton]

To expand/clarify a bit, the example test1.cpp looks like this:

#include "test.h"
#include "singleton.h"

typedef Singleton TestSingleton;

int doTest1()
{
return TestSingleton::instance().doSomething(42);
}

And singleton.h:

template class Singleton
{
public:
static T& instance()
{
static T obj;
return obj;
}
};

So, there is a reference to the static TestSingleton::instance() method, but no symbol for the method, or the TestSingleton::instance()::obj local static declared in the method, appears in the object file. The doTest1 symbol is defined in the object file, as it must be, since the compiler has no way to know whether that symbol is referenced from any other translation unit.

That appears to be at odds with the statement "The symbol in question must be emitted in every TU that uses it...".

If the observed behavior is in fact correct, I'm curious what language in the standard permits it.

Again, I don't have a dog in this fight -- I'm just trying to understand what is going on here:

• Why is the observed behavior different between gcc and clang?
• Is clang's behavior documented anywhere?
• For that matter, is gcc's behavior documented anywhere?
• Which is correct? (Possibly both?)

Thanks!

[rnk]

So, there is a reference to the static TestSingleton::instance() method,
but no symbol for the method, or the TestSingleton::instance()::obj local
static declared in the method, appears in the object file. The doTest1
symbol is defined in the object file, as it must be, since the compiler
has no way to know whether that symbol is referenced from any other
translation unit.

That appears to be at odds with the statement "The symbol in question must
be emitted in every TU that uses it...".

Sorry, I should've elaborated, the "symbols in question" are 'Singleton::instance()' and 'Singleton::instance()::obj'.

Obviously, we're allowed to inline instance() and discard its standalone code after inlining. That's normal and unsurprising, and GCC does it too. It is valid to discard unused code for inline functions because every TU referencing an inline function must emit a copy of its code if it needs it.

What may be surprising is that the same logic applies to the 'obj' variable. After inlining doSomething, we discover that there are no uses of the 'obj' variable, and because we know it came from an inline function, we can drop it. Any other TU that needs 'obj' is responsible for emitting its own copy.

---

While that's all very interesting, I don't think it helps you with your real problem, which is that this singleton pattern isn't working as it should.

My first guess at what's actually happening is that someone in your larger application is using dlopen with RTLD_LOCAL or something similar, and that is preventing the loader from resolving 'Singleton::instance()::obj' to a unique instance. GCC appears to be using a special kind of linkage (the 'u' in the nm output) that punches through RTLD_LOCAL to force global symbol resolution.

... and testing shows that this is probably the case (the final addresses are not equivalent):

$ cat t.cpp
#include <assert.h>
#include <dlfcn.h>
int main() {
void *libt1 = dlopen("./libt1.so", RTLD_LAZY);
assert(libt1);
void (t1)() = (void ()())dlsym(libt1, "t1");
assert(t1);
t1();
void *libt2 = dlopen("./libt2.so", RTLD_LAZY);
assert(libt2);
void (t2)() = (void ()())dlsym(libt2, "t2");
assert(t2);
t2();
}

$ cat t1.cpp
#include <stdio.h>
inline int *singleton() {
static int o;
return &o;
}
extern "C" void t1() { printf("%p\n", singleton()); }

$ cat t2.cpp
#include <stdio.h>
inline int *singleton() {
static int o;
return &o;
}
extern "C" void t2() { printf("%p\n", singleton()); }

$ cat doit.sh
$CXX -g -O2 t1.cpp -fPIC -shared -o libt1.so
$CXX -g -O2 t2.cpp -fPIC -shared -o libt2.so
$CXX -g -O2 t.cpp -o t.exe -ldl
nm libt1.so | grep singleton
nm libt2.so | grep singleton
./t.exe

Different addresses indicate that the singleton pattern failed

$ CXX=clang++ ./doit.sh
000000000020103c V _ZZ9singletonvE1o
000000000020103c V _ZZ9singletonvE1o
0x7fd27240b03c
0x7fd27220903c

Singleton pattern succeeds in G++ even with dlopen craziness

$ CXX=g++ ./doit.sh
000000000020103c u _ZZ9singletonvE1o
000000000020103c u _ZZ9singletonvE1o
0x7fe8dd28c03c
0x7fe8dd28c03c

Switching to RTLD_GLOBAL|RTLD_NOW seems to make the singleton pattern work, even with clang.

OK, now that we know what is going on, we need to figure out what is supposed to happen. It's not clear to me if clang/llvm should use this special GNU unique linkage to force the cross-DSO merging of symbols. I'll have to think about it more.

[llvmbot]

What may be surprising is that the same logic applies to the 'obj' variable.

Surprising is an understatement. Given your description there is no way that any linker, whether ld, ld.so, dlopen(), gold or anything else could possibly resolve the obj variable to the same location in memory -- simply because the symbol does not exist in the object file.

If I understand you correctly, you're saying that this behavior is not a violation of the ODR. I'll have to take your word for that -- I'm not a compiler writer, and trying to understand the ODR from the standard has so far just given me a headache ;-)

I'm also surprised that this hasn't come up before -- the singleton pattern is a fairly common idiom, and so I imagine that there's LOTS of code "in the wild" that use something similar.

In our case, the solution was simple: declare the instance() method in the .h file, but define a specialization in a single .cpp file. This seems to have fixed the problem, but I'm still not convinced that clang's behavior in this scenario is correct.

The discussion of the 'u' linkage is a red herring since the symbol is never placed in the object file in the first place -- no?