Miscompilation on extern(C++) overloads with D types

[dlangBugzillaToGithub]

johanengelen reported this on 2018-07-19T19:17:43Z

Transferred from https://issues.dlang.org/show_bug.cgi?id=19101

CC List

• kinke

Description

The following code is miscompiled:


```
// File a.d
module a;

private struct A {
    int a;
}

extern(C++) int foo(T)(T t) {
    return T.sizeof;
}

int return4() {
    A a;
    return foo(a);
}
```


```
// File b.d
module b;

import a;

private struct A {
    int[100] a;
}

void main() {
    import std.stdio;
    A a;
    writeln(foo(a), " =? ", A.sizeof);
    writeln(return4());
}
```

Compile and run:
> dmd a.d b.d
> ./a
4 =? 400
4

The problem is that in module a, `foo!(a.A)(a.A)` is mangled the same as module b's `foo!(b.A)(b.A)`, because the extern(C++) function name mangler does not use module name prefix for D types. That is, the mangler is mangling `foo!(A)(A)` instead of `foo!(a.A)(a.A)` and `foo!(b.A)(b.A)`. The two function symbols are merged by the linker (instead of erroring on multiple definition), because the symbols come from templates and merging is required behavior.  --> only one of the two _different_ definitions survive, and hence miscompilation results.
The fix: use the full D type for mangling.

[dlangBugzillaToGithub]

kinke commented on 2018-07-30T22:02:25Z

> The fix: use the full D type for mangling.

Assuming you suggest mangling `module mod; extern(C++) class C {};` as `mod::C`, that would prevent all interop with external C++ symbols in the root namespace, as there is no root namespace in D. Likewise, you'd have to have a huge `std.d` file for the std::* symbols etc.

[dlangBugzillaToGithub]

johanengelen commented on 2018-08-14T14:17:55Z

> Assuming you suggest mangling `module mod; extern(C++) class C {};` as `mod::C`

No, of course not.

I suggest mangling `module mod; class C {};` as `mod.C`. Currently we mangle extern(D) types as if they are extern(C++) types for extern(C++) templated functions. In the original example, we currently mangle `foo!(a.A)(a.A)` as `foo!(A)(A)`. Hence the miscompilation / clash with mangling of `foo!(b.A)(b.A)`.

[dlangBugzillaToGithub]

kinke commented on 2018-08-14T16:10:37Z

(In reply to johanengelen from comment #2)
> Currently we mangle extern(D) types as if they are extern(C++) types for 
> extern(C++) templated functions.

Ah now I get it, I overlooked that the 2 structs are extern(D). [The function doesn't need to be templated.] Not sure if a dot can be used in typenames for C++ mangling. Also not sure how much code that would break (structs needing `extern(C++)` for C++ interop, whereas that's optional now).

[dlangBugzillaToGithub]

johanengelen commented on 2018-08-14T16:31:30Z

> Not sure if a dot can be used in typenames for C++ mangling.

Good point. `a.A` should not be mangled as `a::A` (using the C++ namespace divider), because that then would clash with C++ types.

I think it is OK if the mangle can never be expressed in C++ when there are D types involved, because C++ cannot express the D types anyway. What I mean is, `foo!A` should be expressible in C++:
```
extern(C++) struct A {}
extern(C++) int foo(T)(T t) {}
```
but doesn't have to in this code:
```
struct A {}
extern(C++) int foo(T)(T t) {}
```

So C++ doesn't have to be able to express the mangled name, but it would definitely be nice if C++ demanglers can demangle it into something sensible. A dot is actually allowed in symbol names, so we can use it as part of the name? (i.e. pretend that the class name of A is "a.A".)