New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[stable] Fix Issue 19134 - [C++] static const y = new Derived(); ->pointer cast from const(Derived) to immutable(void*)** is not supported at compile time #8533
Conversation
Thanks for your pull request and interest in making D better, @kinke! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.
Please see CONTRIBUTING.md for more information. If you have addressed all reviews or aren't sure how to proceed, don't hesitate to ping us with a simple comment. Bugzilla references
Testing this PR locallyIf you don't have a local development environment setup, you can use Digger to test this PR: dub fetch digger
dub run digger -- build "stable + dmd#8533" |
src/dmd/expressionsem.d
Outdated
// skip the vptr assignment during CTFE; it's not supported and superfluous, | ||
// as an external super in C++ is non-CTFE-able anyway | ||
// => __ctfe || cast(void) (this.__vptr = __vtbl) | ||
ate = new LogicalExp(loc, TOK.orOr, new IdentifierExp(loc, Id.ctfe), new CastExp(loc, ate, Type.tvoid)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel like a potentially superior implementation would be to detect if super()
is extern, and then skip the assignment on that basis. That way, even outside of CTFE, the redundant work to assign the vtable will be skipped in cases when it doesn't need to.
test/runnable/cppa.d
Outdated
{ | ||
int a = 123; | ||
this() { a += 42; } | ||
void foo() {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you make foo
return a
and call it from the tests ?
Possible to tweak this in the way I suggest to gain the runtime benefit also? |
Thanks Manu, that's the better approach indeed. Implemented now and vptr test added. |
This LGTM, given that we agree on the theory as I suggested... In regular D, during the entire constructor chain, the vtable is set to the most-derived vtable. It occurs to me that, any extern(C++) constructor won't be able to determine whether the most-derived vtable is applied, of some super-class vtable is applied from a C++ base constructor. So, what I'm actually thinking would be better is to revert this to your former solution, and instead implement logic that matches the C++ semantic; that is, any extern(C++) constructor will assign the vtable pointer for that type on function entry. I think the most reasonable thing to do here with vtable assignment semantics is to match the C++ rules, since we can't affect that behaviour when linking externally. That said; that puts us back into the exact same situation where the vtable assignment on function entry may not CTFE? Why is that an issue? Can that be fixed some other way? |
...or just merge this patch :) ... it's probably fine! |
Well, let's say, do merge this patch (it's superior to the existing code), and then the logic I detail above may be implemented in a follow-up, if someone wants to do it. |
I think what happens during extern(C++) construction is this:
So by restoring the vptr after each external super() call (possibly implemented in C++), we should be fine. |
That's correct, but we don't restore it to the most-derived pointer (ie, the original pointer), we restore it to the vptr for the type at the level of the constructor being executed... In fact... this won't work! With this logic, the most-derived vtable will never be restored if there is a D super-class. >_< Is it possible to restore it to the most-derived ptr? I think we would need to take a local copy prior to calling an extern-super, and then restore it after. |
Yep you're right, a copy seems necessary. |
Yeah. Let's do that. |
And it only works because C++ sets the vptr after calling its super and right before the actual ctor body, right? Otherwise, extern(C++) base classes above an actual C++ one may not have the final vptr set when entering their ctor. |
So #8362 introduced this breaking change for ctors in |
Yes.
Right. If there's an actual C++ ctor in the stack, that call enters with the most-derived vptr, then before it assigns its local vptr, it will call the further super(). If THAT super() is a D function again, it will be entered still with the most-derived vptr. The callstack will always reach the base ctor with the most-derived vptr assigned.
I think we actually got confused somewhere along the way. Our original intent was to preserve D semantics in the D constructors (which depends on this fix that we record and restore the vptr around calls to an extern But IIRC, there was something that convinced us we need to go with C++ semantics all the way down the tree, and so we implemented it the way it is now. I think the reason we decided that the D ctor should match C++ semantics was that, in C++, override functions can be written to EXPECT that the class is constructed when they're called. Imagine a D base class calls a virtual implemented in C++, which expects that it is constructed, but because it was called from a D base class with the most-derived vtbl assigned, that assumption that the C++ code may make is not reliable. |
Of course, that's a very unlikely case, and it's a value-judgement. It may be a better call to preserve D semantics such that D code works as expected. It's strongly discouraged to call virtuals from ctors in D for this exact reason... so calls to unsuspecting C++ overrides are likely to be mitigated by that recommendation. |
I tend to agree, otherwise the language spec would get too hairy. It's also clear that calling the final overrides in a C++ base ctor would be a very bad idea, as the fields of more derived types are most likely not initialized yet (garbage). In an extern(C++) one implemented in D, all fields are/should at least be initialized according to the init symbol (and possibly already mutated earlier in more derived ctors, by delaying the super() call in those). Related to this is the question of how to deal with this init-symbol-blit dependency of the extern(C++) ctor chain. In order to make extern(C++) classes seamlessly usable in C++ (e.g., allocate+construct via C++ new without ending up with garbage due to untouched fields normally initialized by the blit), we could blit the current class' fields from the init symbol (of the current class; these initializer fields should perfectly match the most derived class' init symbol) upon ctor entry. [This blit would need to be skipped during CTFE too.] |
I agree, we need to init when calling D ctor from C++... [thoughts below]
No, in CTFE, this is already current behaviour for construction of any class type. No change is required for CTFE. I don't think ANY of this C++ logic is involved in CTFE, because it's all related to external linkage, and CTFE can't resolve if there's any extern calls in the chain anyway.
So, we're dealing with the situation when an extern(C++) ctor is defined in D, and NOT in C++; that is, when C++ constructs, it will call the D function. In D, prior to construction, init will be assigned. C++ doesn't execute such logic. I think the proper solution is this:
When C++ constructs a class, the generated __cppctor() that is mangled for C++ first performs the init assignment, which properly prepares for construction, then calls into the regular ctor call tree. I think this works elegantly. So there are 2 pieces of special-sauce in the end:
I think that resolves all the edge cases? |
Note: of course, D would never call this magic __cppctor(), since the init assignment is handled by the language, which means there is no interaction with CTFE in this solution. |
// D
extern(C++) class A
{
int a = 1;
this() {}
void virtualFoo() {}
}
// C++
class B : public A
{
int b = 2;
B() : A() {} // => calls the 'actual' C++ ctor of A, performing the A init blit
};
// D
extern(C++) class C : B
{
int c = 3;
this()
{
a = 666;
super();
}
}
|
Good point. There is an edge case when C++ derives from D... |
Another issue with this: extern(C++) class A { int a = 123; } This would now need a C++-mangled ctor (just blitting), although it has no regular ctor. |
Yeah, that __cppctor() symbol I described should probably be present for every ctor implemented in D (including the default ctor or default init). |
I think this motivates leaning back towards extern(C++) ctors matching C++ construction semantics with vptr assignment at entry. |
Yeah that's what I was thinking too, but I'd go further and get rid of the D pre-init blit for extern(C++) classes altogether (and so no need for extra __cppctor complexity). On extern(C++) ctor entry, call super() (implicitly or check if first statement) and then blit vptr & class-specific fields from init symbol; disallow later explicit super() calls; no vptr restoring necessary. And add an implicit default ctor in case there's no explicit one. |
That sounds comprehensive, do you know how to implement all that? I'd suggest that this PR should be merged for now to restore CTFE, and that complex patch you describe can be a follow up... |
Of course not, but it should be doable. @rainers might want to help out too. ;)
Yeah, we need a short-term solution, but I should add the copy and so restore the proper vptr. With this in its current state, if we have a C++ base class A and two derived extern(C++) classes |
Yes, exactly. |
Oh man, just when I thought I had it (works fine here with LDC on Win64), the next problem arises - just like for dtors, there appear to be multiple C++ ctors as well ( |
Do you think it's wise to test/prove that most-derived vptr is present like this, since the result of our conversation above, was that we're gonna change it to strictly conform with C++ semantics anyway throughout the stack? :) |
The test has shown 2 Posix issues (extra ctors & C++ classes not constructible with D |
14c4361
to
ca2d895
Compare
|
||
auto restoreVptr = new AssignExp(loc, vptr.syntaxCopy(), new VarExp(loc, vptrTmpDecl)); | ||
|
||
Expression e = new CommaExp(loc, declareTmps, new CommaExp(loc, restoreVptr, new VarExp(loc, superTmpDecl))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When using Expression.combine(a, b, c, d)
here, creating equivalent ((a, b), (c, d))
, all hell breaks loose, requiring additional expressionSemantic()
calls; combine(lhs, rhs)
additionally sets the new CommaExp.type
to the rhs type, which seems to make expressionSemantic()
not descend to both children (which then leads to forward-referencing errors later on...).
src/dmd/expressionsem.d
Outdated
auto pte = new DotIdExp(loc, new ThisExp(loc), Id.__vptr); | ||
auto ate = new AssignExp(loc, pte, ase); | ||
auto vptr = new DotIdExp(loc, new ThisExp(loc), Id.__vptr); | ||
auto vptrTmpDecl = copyToTemp(0, "__vptrTmp", vptr); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These tmp names are uniqued anyway, so no need for the local superid
counter (and there should be a single super() call only).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good.
|
||
auto declareTmps = new CommaExp(loc, declareVptrTmp, declareSuperTmp); | ||
|
||
auto restoreVptr = new AssignExp(loc, vptr.syntaxCopy(), new VarExp(loc, vptrTmpDecl)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if vptr.syntaxCopy()
is required or whether vptr
could be reused directly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Better safe than sorry, using the same expression instance twice might cause issues if the AST gets modified by lowerings.
sprintf(buf.ptr, "__super%d", superid++); | ||
auto tmp = copyToTemp(0, buf.ptr, result); | ||
// so we have to restore it, but still return 'this' from super() call: | ||
// (auto __vptrTmp = this.__vptr, auto __superTmp = super()), (this.__vptr = __vptrTmp, __superTmp) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I might have missed some comments, but is saving and restoring the vptr good enough? If the class instance is created in C++, the vptr might be garbage. Granted, other fields might not be initialized, too, but that can be solved in user code.
Can the CTFE issue be solved by preassigning some appropriate type to one of the expressions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO, it's fine for now and reverts to previous D vptr semantics in extern(C++)
ctors. IIRC, I didn't find anything about this breaking change mentioned in the changelog.
but that can be solved in user code.
I don't see a good way of doing that (and don't want to burden the user with these details), and so suggested #8533 (comment) for fixing this preblit-dependency properly in the long run. [The implicit default C++ ctor would be useful for extern(C++)
structs too btw.]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking of initializing the fields explicitly with some foreach over C.tupleof, but that is actually putting the burden on the user.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO, it's fine for now and reverts to previous D vptr semantics in extern(C++) ctors.
Ok, allocating a class defined in D from C++ is still trouble anyway, and solving this for classes that have super() calls into C++-code is a very special case, only.
|
||
// TODO: Allocating + constructing a C++ class with the D GC is not | ||
// supported on Posix. The returned pointer (probably from C++ ctor) | ||
// seems to be an offset and not the actual object address. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some platforms don't return the this pointer from the C++ constructor, see scopeAllocCpp
above for a workaround.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I already fixed this for LDC, by simply ignoring the returned pointer, see ldc-developers/ldc@25d7a79.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess dmd should do the same.
|
||
// TODO: Allocating + constructing a C++ class with the D GC is not | ||
// supported on Posix. The returned pointer (probably from C++ ctor) | ||
// seems to be an offset and not the actual object address. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess dmd should do the same.
|
||
auto declareTmps = new CommaExp(loc, declareVptrTmp, declareSuperTmp); | ||
|
||
auto restoreVptr = new AssignExp(loc, vptr.syntaxCopy(), new VarExp(loc, vptrTmpDecl)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Better safe than sorry, using the same expression instance twice might cause issues if the AST gets modified by lowerings.
src/dmd/expressionsem.d
Outdated
auto pte = new DotIdExp(loc, new ThisExp(loc), Id.__vptr); | ||
auto ate = new AssignExp(loc, pte, ase); | ||
auto vptr = new DotIdExp(loc, new ThisExp(loc), Id.__vptr); | ||
auto vptrTmpDecl = copyToTemp(0, "__vptrTmp", vptr); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good.
sprintf(buf.ptr, "__super%d", superid++); | ||
auto tmp = copyToTemp(0, buf.ptr, result); | ||
// so we have to restore it, but still return 'this' from super() call: | ||
// (auto __vptrTmp = this.__vptr, auto __superTmp = super()), (this.__vptr = __vptrTmp, __superTmp) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking of initializing the fields explicitly with some foreach over C.tupleof, but that is actually putting the burden on the user.
sprintf(buf.ptr, "__super%d", superid++); | ||
auto tmp = copyToTemp(0, buf.ptr, result); | ||
// so we have to restore it, but still return 'this' from super() call: | ||
// (auto __vptrTmp = this.__vptr, auto __superTmp = super()), (this.__vptr = __vptrTmp, __superTmp) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO, it's fine for now and reverts to previous D vptr semantics in extern(C++) ctors.
Ok, allocating a class defined in D from C++ is still trouble anyway, and solving this for classes that have super() calls into C++-code is a very special case, only.
b0dca8e
to
44d70c2
Compare
And revert to D vptr semantics in constructors of extern(C++) classes, i.e., call the most derived override in base constructors. 2.081 introduced a breaking change here, using the most derived override before the super() call and the current class' afterwards. The proper mid-term solution is probably to emit fully C++-compatible constructors for extern(C++) classes, calling the super ctor right at the start and then blitting vptr and class-specific fields from the init symbol. extern(C++) classes (and structs) should get an implicit default ctor (only if there's no other ctor in the class case). extern(C++) classes could then be allocated and properly constructed via C++ `new`; struct declarations in C++ headers for extern(C++) structs wouldn't need to duplicate the field initializers and could simply declare the default ctor (emitted by the D compiler and blitting the init symbol).
Rebased to retrigger the tests, squashed and commit msg added, incl. a sketch for a proper solution. - This should make it into 2.082 IMO, it's a regression fix after all. |
You're super awesome for wrangling this beast!! Thanks again so much! |
No description provided.