-
-
Notifications
You must be signed in to change notification settings - Fork 606
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix issue 19278 - extern(C++, "name") doesn't accept expressions #8787
Conversation
Thanks for your pull request and interest in making D better, @TurkeyMan! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.
Please see CONTRIBUTING.md for more information. If you have addressed all reviews or aren't sure how to proceed, don't hesitate to ping us with a simple comment. Bugzilla references
Testing this PR locallyIf you don't have a local development environment setup, you can use Digger to test this PR: dub fetch digger
dub run digger -- build "master + dmd#8787" |
return; | ||
} | ||
ns.ident = Identifier.idPool(ident); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can I get guidance on this block? I'm not sure where/how it should be placed inside this function... and is there anything that should be done after the late-assignment of the symbol name that might have been missed prior?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please spilt that loop into the else case followed by a loop over the rest of the namespaces.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, the AST at the start looks like
NSpace foo::bar::baz
+- Dsymbols members
After the end of the above it should look like (modulo s.addMember(sc, sc.scopesym)
)
NSpace foo
+-Dsymbols members
+- NSpace bar
+- NSpace baz
and you want it to end up looking like
NSpace foo
+- NSpace bar
+- NSpace baz
+- Dsymbols members
In which case instead of that I'd validate all the namespace identifiers first, and then pull the ol' switcharoo:
Enclosing Scope (e.g. module)
+-NSpace foo::bar::baz
+- Dsymbols members
split, do not insert into enclosing scope ->
NSpace bar::baz (parentless)
Enclosing Scope
+- NSpace foo
+- Dsymbols members
expand ->
NSpace bar (parentless)
+- NSpace baz
Enclosing Scope
+- NSpace foo
+- Dsymbols members
rotate reassign ident
s ->
NSpace foo (parentless)
+- NSpace bar
Enclosing Scope
+- NSpace baz
+- Dsymbols members
pull the switcharoo (reparenting insert) ->
Enclosing Scope
+- NSpace foo
+- NSpace bar
+-NSpace baz
+- Dsymbols members
This means that the original namespace keeps all the members and the only reference to the original (because its a tree), from the enclosing scope, is overwritten. then you can addMember
the intervening namespaces.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right... see, logically it makes sense... but i have no idea what the code that implements that might look like.
Those functions addMember
, setScope
, importAll
, and what they mean relative to the 'current' scope (which is a concept that still isn't clear), and when they should all be called... and why. I can't imagine how to implement that sequence of operations with the primitives at hand.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure, try asking on slack.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please feel free to submit a patch if you think you know how to implement this logic ;)
01a6512
to
0529127
Compare
0529127
to
96828a7
Compare
9ced37c
to
45d731d
Compare
Does this mean #8781 is abandoned? Should it be Phantom Zoned, or just closed? |
45d731d
to
583c9f3
Compare
No, I think that is complementary; being able to compose strings this way is a lot less useful if you can't compose a string with I'll merge them if it's accepted, but I wanted to review the detail in this one as a self-contained patch, since it's more complex. I didn't want to muddy the waters. |
I think strings with "::" in them feels more uniform with |
If you can comment on the work-block; is it in the right location? Ordered correctly relative to the other work? Does it need any further semantic work? |
The reason I suggested several arguments/tuple was to avoid the need to decide between But I didn’t think of a need to compose the string literal. |
src/dmd/parse.d
Outdated
{ | ||
nextToken(); | ||
cppMangleOnly = true; | ||
identExp = parseCondExp(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My parser knowledge is a bit limited, but shouldn't this be an assignExp?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How would an assign expression be valid here? I think ?:
is the highest level grammar applicable (and likely to occur) in this context...
Fair enough. That was one of primary motives for making it a string ;) So, I'm feeling '::' because that's a cut & paste of the C++ namespace that it's linking against, and it also leaves no room for confusion that it might be a D identifier.
It's one small loop; fortunately D is astonishingly good at string processing! Splitting on '::' is not trouble. |
I know it's not technically difficult, it just feels wrong. |
I agree, it's hackish to have a "language within a language". |
Ummm. C++ support is definitely a language within a language to some quite ambitious extent... this expression describes a C++ namespace for mangling purposes, and that should be a string such that it may be any valid C++ namespace. The only grammatical way to express a C++ namespace in D is within a string, and as a bonus by doing so, we're not leaking any possibility for misconception that the thing is anything other than a C++ namespace to the user... I think it's perfect. |
Maybe I didn't catch the full discussion, but why is the parenthesis required? |
It's not required... any expression that doesn't start with an identifier is unambiguous... it's just that i figured requiring parens was a little more explicit and less susceptible to surprise. |
It just looks/looked like an arbitrary limitation to me and if I would look at code that uses it I would probably wonder why the author used the unnecessary parenthesis. {a: b} = {a: 42};
// b is 42
let obj = {};
({a: obj.b} = {a: 12});
// obj now has a member b with the value 12 Anyhow, AFAICT that's not the main concern here. |
There's a potential ambiguity between the named scope and a string expression. |
It is D syntax. I'm mean, it is D after all. |
src/dmd/dsymbolsem.d
Outdated
parentns.members.push(childns); | ||
|
||
// not sure how to fix up... | ||
// there is symtab, _scope is all wrong... etc. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what to write here to fix up the parent/scope pointers...
96e3f5b
to
8aa7fcb
Compare
No... it's a string, that contains a C++ identifier to be used for C++ mangling. The D syntax is that it accepts a string of a valid C++ identifier for mangling. It's not valid D identifiers, and it shouldn't be possible to confuse it as such. |
test/fail_compilation/cppmangle.d
Outdated
fail_compilation/cppmangle.d(10): Error: invalid zero length C++ namespace | ||
fail_compilation/cppmangle.d(14): Error: expected valid identifer for C++ namespace but got `0num` | ||
fail_compilation/cppmangle.d(18): Error: string expected following `,` for C++ namespace, not `)` | ||
fail_compilation/cppmangle.d(10): Error: string expected following `,` for C++ namespace, not `)` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to update the line number and message, from the autotester:
expected:
----
fail_compilation/cppmangle.d(10): Error: string expected following `,` for C++ namespace, not `)`
----
actual:
----
fail_compilation/cppmangle.d(8): Error: found `,` when expecting `)`
fail_compilation/cppmangle.d(8): Error: declaration expected, not `)`
----
so this would become
fail_compilation/cppmangle.d(9): Error: found `,` when expecting `)`
fail_compilation/cppmangle.d(9): Error: declaration expected, not `)`
Walter accepted the previous PR before it was finished baking. I'm finishing it up. |
Can we deprecate |
Since they are just
True for client code, but truly manageable for the most common std types. I'm not attempting to block this PR, but I would prefer us to gather experience before we put in changes to the syntax that don't enable any new usage. |
Technically GitHub does not recognize |
Only one of my editors does it right.
There's almost no
Client code is what matters! And I'd argue that STL is one of the worst offenders; nobody knows what members they have. In C++, I would say the STL is my single greatest use for auto-complete assistance. Nobody knows how to work that thing, it's all weirdly named functions and iterators and rubbish.
This enables one of the key cases that it was intended for, and the only case I'm aware of that's in development for druntime. I'm just finishing the patch that was already merged, I see this is a completion of that task. I never imagined for a second that string expressions wouldn't be There's definitely a future where named scopes are granted their own syntax, and the disambiguation requirement ceases to exist. In 99% of cases, just string literals are fine, and no disambiguation is required. It's really just the STL, and one other private project that I've encountered where an expression is necessary. It's not pervasive. |
We going to need a DIP for that to happen. I can write up a quick draft on it. |
Sure. But I don't think it's important right now. I doubt anybody in the world is waiting on that. |
Ping @WalterBright |
src/dmd/dsymbolsem.d
Outdated
TupleExp tup = name ? null : resolved.toTupleExp(); | ||
if (!tup && !name) | ||
{ | ||
ns.error("expected string expression for namespace name, got `%s`", ns.identExp.toChars()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
... or tuple
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The tuple must be strings. I considered this message for a while, but felt that sticking with this message was less confusing. The message is: this expects strings. To explain that it must be a string or a tuple of strings felt too long and potentially more confusing that this message. YMMV?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Like, the precision makes sense to a compiler author, but a user just needs to know that they need to supply strings.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So maybe use the plural "string expressions"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To explain that it must be a string or a tuple of strings felt too long and potentially more confusing that this message
In theory it could be an array of strings instead of a tuple, which is not the same thing.
src/dmd/dsymbolsem.d
Outdated
const(char)[] ident = name.toStringz(); | ||
if (ident.length == 0 || !Identifier.isValidIdentifier(ident)) | ||
{ | ||
ns.error("expected valid identifer for C++ namespace but got `%.*s`", cast(int)ident.length, ident.ptr); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As can be seen in the failure tests, this yields "namespace __anonymous
" which is confusing. Better just call the global error function to omit it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How do I do that? Just error()
? That's fine to do here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, error(ns.Loc, ...)
should work.
src/dmd/dsymbolsem.d
Outdated
// resolve the namespace identifier | ||
sc = sc.startCTFE(); | ||
Expression resolved = ns.identExp.expressionSemantic(sc); | ||
resolved = resolved.expressionSemantic(sc); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it really necessary to call expressionSemantic again here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, good catch! This must be a cut&paste fail.
return; | ||
} | ||
const(char)[] ident = name.toStringz(); | ||
if (ident.length == 0 || !Identifier.isValidIdentifier(ident)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
D allows more characters in identifiers as C++, but that's probably ok.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I thought about that, but I felt that writing a new validation function that's used in exactly one place was overkill.
You're limited to C++ subset in practice anyway, because the C++ code is restricted by its rules.
src/dmd/dsymbolsem.d
Outdated
if (i > 0) | ||
{ | ||
// insert the new namespace | ||
Nspace childns = new Nspace(parentns.loc, Identifier.idPool(ident), null, parentns.members, parentns.mangleOnly); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
parentns.loc
and parentns.mangleOnly
are constant here, maybe that's more obvious by using ns
instead of parentns
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay.
@@ -2841,6 +2902,11 @@ private extern(C++) final class DsymbolSemanticVisitor : Visitor | |||
sc.parent = ns; | |||
foreach (s; *ns.members) | |||
{ | |||
if (repopulateMembers) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this necessary? The scopesym should not change by inserting mangleOnly namespaces and it seems dangerous to insert symbols into the symbol table again. Maybe calling s.setScope(sc) is enough.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't understand this code as well as I'd like. I can try your suggestion?
I'd love an expert to fix these couple of lines, they made me a touch nervous, but the unit tests all passed, so I had moderate confidence I worked it out right...
src/dmd/parse.d
Outdated
} | ||
} | ||
else | ||
else // extern(C++, [StringExp]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The brackets don't match the comment before the function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mmm, WIP comment...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You might also want to add the extern(C++,struct)
to the comment.
@@ -15,9 +16,10 @@ extern(C++, "0num") | |||
{ | |||
} | |||
|
|||
extern(C++, "std", ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this no longer an error or can the test be kept?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not a meaningful test anymore. This was previously a specific case that the parser implemented manual handling for (hence the tests). Since I changed the parse code to use ParseCondExp()
, this testing for no expression/fail commas is handled and tested elsewhere.
These explicit tests no longer prove anything.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, but it might be interesting to verify whether the tuple is allowed to have a trailing comma or not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed with Rainers here, can you add a trailing comma somewhere in your cppmangle
tests ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you sure? The fail is in the parser, not semantic... so it can't be in this file with the rest of these semantic errors. I need to add a whole new file to the tester just to test that one thing, which should be tested by whatever already tests parseCondExp()
...
I can do it, I just don't know why, and I have to create a whole new file just for that one test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hum, right, because you don't actually pass a tuple, so the point in moot. I guess when we fix this code (I see this state as temporary), we'll fix it. At the moment basically any usage of the feature needs to be surrounded by parenthesis.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are expressions that aren't ambiguous with an identifier.
extern(C++, "hello" ~ "world")
doesn't need parens for instance.
fail_compilation/ice17074.d(29): Error: declaration expected, not `)` | ||
--- | ||
*/ | ||
extern(C++, ...) void ice_token(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are these tests change or removed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as above; they were cases that were manually handled, I replaced with ParseCondExp()
and evaluate with CTFE; the testing for invalid expressions is comprehensively tested elsewhere.
I see no reason to increasing test time for arbitrary invalid expressions which are already very well tested.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, shouldn't have an effect on coverage.
*/ | ||
Expression identExp; | ||
|
||
extern (D) this(const ref Loc loc, Identifier ident, Expression identExp, Dsymbols* members, bool mangleOnly) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Having both ident
and identExp
is redundant with mangleOnly
. I wonder if it would be better to always parse expressions and determine in semantic whether there is a single fully qualified name. That would lift a couple of restrictions with respect to what expressions are detected correctly, but it's also a larger change to the existing case, so maybe left for a separate refactoring.
Also needs a rebase |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you:
- rebase
- squash
- provide a more sensible title to the resulting commit, e.g.
Fix issue 19278 - extern(C++, "name") doesn't accept expressions
- Ping me when it's done
Tested it locally, it LGTM, and if it's the last thing needed for proper stdlib support, we might as well get it in at the beginning of the new release.
{ | ||
void test19278_4(); | ||
} | ||
version(Win64) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can I haz POSIX tests ?
@@ -15,9 +16,10 @@ extern(C++, "0num") | |||
{ | |||
} | |||
|
|||
extern(C++, "std", ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed with Rainers here, can you add a trailing comma somewhere in your cppmangle
tests ?
ee61f98
to
7e07f60
Compare
7e07f60
to
db36d97
Compare
Wtf is with semaphore, and why doesn't it ever work? The error is unhelpful. |
Ignore it, its unrelated. |
Are we done here? |
Just squash once more and I think thats it. |
extern(C++, Expression)
Accept an expression, attempt to resolve in semantic.
Ambiguity with scoped namespace identifier can be defeated with parens.