Deprecate extern(C++, identifier)

[Geod24]

Based on #10021 . Only the last commit is should actually be part of this PR.

This features tries to import C++ semantics into D, by giving namespaces a scope.
However, when it comes to interfacing with C++,	we only care about namespaces
in the context of mangling: they shouldn't be used for code organization.

Namespaces don't work so well for D: for example, namespaces in C++ can be spread across multiple files,
which is something D can't and won't do because it conflicts with its very core, modules.
It also	leads to issues	where identifiers used for namespaces
(which are usually outside of the programmer's control)	end up conflicting with	user code.

Instead, users should use `extern(C++, "string")`,
which makes D code follows D semantic while letting users link with a symbol in a namespace.

[Geod24]

CC @TurkeyMan

[TurkeyMan]

I don't know that Walter's initial implementation is something that people want or not... I mean, I don't use it, but I can only speak for myself on that front.

I support this though; I think if we're going to have named scopes, then we should reintroduce the feature as a language feature in its own right. names scopes shouldn't be conflated with extern(C++).

[thewilsonator]

is pastMixinAndNspace even used anymore, it can probably just be removed.

[Geod24]

Actually most of the change related to toParent3 can be reverted.
That's quite a lot of changes though, I was considering putting them in an extra commit, what do you think ?

[dlang-bot]

Thanks for your pull request, @Geod24!

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub fetch digger
dub run digger -- build "master + dmd#10031"

[WalterBright]

No. We've hashed this out endlessly before. If you don't like it, you don't have to use it.

[Geod24]

@WalterBright : Do you actually use this ?

[atilaneves]

@Geod24 I might have to for dpp; I'm not sure yet. It's complicated. In any case, since I'm not sure I'm firmly against this because it might make C++ interop harder.

[SSoulaimane]

The problem is this behavior is neither fully C++ nor fully D, if you want to keep namespaces as scopes then make them cross modular like C++ because that's how people will declare and use them when interfacing with C++ code. Half baked features tend to cause more trouble than they solve.
Unless I am missing something, the current guideline is to use modules. Then if one should use modules, namespaces should be deprecated.

[WalterBright]

Do you actually use this ?

I've used it when interfacing with C++. I don't deal with C++ the way Manu does.

if we're going to have named scopes, then we should reintroduce the feature as a language feature in its own right. names scopes shouldn't be conflated with extern(C++).

It was done that way to suggest it is for interfacing to C++.

Half baked

It's not half baked. The half-baked method is the string one, because it's an ugly hack. For one thing, it introduces a scope and then totally ignores it.

[WalterBright]

As I've pointed out on the n.g., the string version is unprincipled and for people who want to organized D modules in an unprincipled manner like C++ code often is. I can't agree that method should be preferred, nor agree that the principled method should be removed.

[SSoulaimane]

• It is not necessary for interfacing C++, no one is demanding it.
• It doesn't fully work like C++ namespaces
• the whole namespace paradigm doesn't work well with modules

So it is half baked. The more you invest in it the more trouble it makes, that's why people should not use it at all.

it introduces a scope and then totally ignores it.

That was fixed already. #10021.

[SSoulaimane]

As I've pointed out on the n.g., the string version is unprincipled

I'm not sure I understood what you mean there.

[WalterBright]

That was fixed already. #10021.

The description says it is a refactoring. Really, we should not be calling PRs "refactoring" if they have wholesale semantic changes.

[WalterBright]

I'm not sure I understood what you mean there.

Introducing a scope without introducing a scope is unprincipled. Using strings instead of identifiers is ugly. I don't see where the refactoring introduces a scope. In fact, I don't understand the point of the refactoring.

[SSoulaimane]

Introducing a scope without introducing a scope is unprincipled

That what #10021 changed, it doesn't introduce a scope anymore. extern(C++, "N") now compiles similar to extern(C++), it's just a subclass of AttribDeclaration, so we can say it is more principled now.

I don't see where the refactoring introduces a scope

Sorry I wasn't clear, it avoids the old behavior of discarding a scope by not introducing the scope.

Using strings instead of identifiers is ugly

It just happened that the identifier version also introduces a scope. Maybe the deprecation could be put on N.member where N is a namespace i.e when a symbol is qualified by its namespace. This way the identifier syntax could be preserved.

[WalterBright]

Since the ident scope behavior is still there, this PR has accomplished nothing that I can discern.

so we can it is more principled now.

It's unprincipled because it tries to emulate a scope on the C++ side without a scope. If you are dealing with nested/multiple name spaces on the C++ side, that doesn't work on the D side, because the names are not distinguishable. The string version just dumps everything into the D global name space.

It's like C macros. No scoping. It's unprincipled.

[WalterBright]

It just happened that the identifier version also introduces a scope

That didn't "just happen". It was deliberately designed that way.

[SSoulaimane]

It just happened that the identifier version also introduces a scope

That didn't "just happen". It was deliberately designed that way.

That's not what I meant, I meant the intention was not to undermine the syntax per se but the associated behavior which just happened that way there no reason why the identifier syntax has to introduce a scope.

[SSoulaimane]

The recommendation is to use modules and only modules. There is no concrete support of C++ namespaces i.e. cross files and everything. So if one must group extern(C++) declarations they should use the means available in D, not something that looks like a C++ namespace but in realty is just like using a struct with the same rules and caveats.

[WalterBright]

but the associated behavior which just happened that way there no reason why the identifier syntax has to introduce a scope

There was definitely a reason, which is why I deliberately implemented it that way. Here's the reason:

Namespaces in C++ introduce a scope

[SSoulaimane]

but the associated behavior which just happened that way there no reason why the identifier syntax has to introduce a scope

There was definitely a reason, which is why I deliberately implemented it that way. Here's the reason:

Namespaces in C++ introduce a scope

Nothing to do with the syntax.

[WalterBright]

in realty is just like using a struct with the same rules and caveats.

It is not or I would have implemented it as a struct. Please understand that I've implemented a C++ compiler, I know how namespaces work. By removing the scope on the D side, you cannot have the same name in two C++ namespaces.

The POINT of namespaces in C++ is to have scoped lookup.

[SSoulaimane]

The POINT of namespaces in C++ is to have scoped lookup.

That's why D has modules, and this is why D doesn't have namespaces like C++. This feature was supposed to interface with C++ in an idiomatic D manner, not bringing C++ quirks into D, and in this case even with new quirks.

[SSoulaimane]

Please understand that I've implemented a C++ compiler, I know how namespaces work. By removing the scope on the D side, you cannot have the same name in two C++ namespaces.

You can have the same name in two modules, and this is the recommendation anyway.

[WalterBright]

You can have the same name in two

Nope:

extern (C++, "A") { int foo() { return 1; } }
extern (C++, "B") { int foo() { return 2; } }
static assert(foo() == 1);

foo.d(4): Error: foo.foo called with argument types () matches both:
foo.d(2):     foo.foo()
and:
foo.d(3):     foo.foo()
foo.d(4):        while evaluating: static assert(foo() == 1)

[WalterBright]

This feature was supposed to interface with C++ in an idiomatic D manner, not bringing C++ quirks into D, and in this case even with new quirks.

extern (C++, A) {
    int foo() { return 1; }
    extern (C++, B) {
        int foo() { return 2; }
        static assert(foo() == 2);
        static assert(A.foo() == 1);
    }
    static assert(foo() == 1);
}
static assert(A.foo() == 1);
static assert(A.B.foo() == 2);

This behavior is the same as on the C++ side. The fact that C++ namespaces introduce a scope is the entire point of them, not a quirk. The above cannot be made to work with the string method.

[TurkeyMan]

Let me fix your example:

pragma(mangle, "wow") int foo() { return 1; }
pragma(mangle, "omg") int foo() { return 2; }
static assert(foo() == 1);

Nothing about extern(C++) should make this code valid.

If you want a feature to introduce a named scope, then make a feature to introduce a named scope. That still doesn't specifically have anything to do with extern(C++), but it might have an interaction the way you'd like.

[WalterBright]

Nothing about extern(C++) should make this code valid.

C++ namespaces introduce a scope. I know you treat all C++ namespaces as being in the same scope, and never write the same name in different namespaces, but you can't generalize that to all C++ programmers. It's not a quirk nor an accident nor my delusion that C++ namespaces introduce a scope. It is that way so the names don't collide.

The way you use C++ namespaces might as well be the C way of std_name rather than std::name. It's not that your practice is necessarily wrong, but what is wrong is insisting that D's C++ interface only work with your style of programming.

Insisting that C++ namespaces are mangling only and have no semantic consequences and do not introduce a scope is just plain wrong. It's an objective fact, not my opinion.

it might have an interaction the way you'd like.

It is not what I'd "like". It is the way C++ works.

[Geod24]

@Geod24 I might have to for dpp; I'm not sure yet. It's complicated.

If you could expand on the situation where you see this feature as potentially beneficial, I'd be glad to hear it.

In any case, since I'm not sure I'm firmly against this because it might make C++ interop harder.

Fair enough. Any idea when you will have a clearer picture ?

Insisting that C++ namespaces are mangling only and have no semantic consequences and do not introduce a scope is just plain wrong. It's an objective fact, not my opinion.

An objective fact no one but you seem to agree on.

Namespaces in C++ can be cross files. Sometimes they serve the purpose that module serve, and for deeply nested one, they serve the purpose of packages.
If C++ namespace semantic was fully imported in D, the following code could be possible:

--- a.d
module a;
extern(C++, namespace) void foo(int);
--- b.d
module b;
extern(C++, namespace) void bar(const char*);
--- main.d
module main;
import a, b;
void main ()
{
    namespace.bar("Hello world".ptr);
    namespace.foo(42);
}

At the moment, this errors out:

main.d(7): Error: a.namespace at a.d(2) conflicts with b.namespace at b.d(2)
main.d(7): Error: no property bar for type void
main.d(8): Error: a.namespace at a.d(2) conflicts with b.namespace at b.d(2)

Of course there is a few simple ways out of this:

module main;

import a : foo;
import b : bar;

void main ()
{
    bar("Hello world".ptr);
    foo(42);
}

Which is just... Removing the scope introduced by extern(C++, namespace), essentially, and keeping the mangling. Well if I want to qualify them I can just used static import right ?

module main;

static import a;
static import b;

void main ()
{
    b.namespace.bar("Hello world".ptr);
    a.namespace.foo(42);
    // But what's the point of using `x.namespace` when `x` is enough... ?
    b.bar("Hello world".ptr);
    a.foo(42);
}

What if instead my namespaces are declared inside my current module ?

module main;

extern(C++, namespace) void foo(int);
extern(C++, namespace) void bar(const char*);

void main ()
{
    namespace.bar("Hello world".ptr);
    namespace.foo(42);
}

Well that's not possible either:

main.d(4): Error: namespace `main.namespace` conflicts with namespace main.namespace at main.d(3)

So namespace is ambiguous... But what if I remove the namespace qualifier at the call site?

module main;

extern(C++, namespace) void foo(int);
extern(C++, namespace) void bar(const char*);

void main ()
{
    bar("Hello world".ptr);
    foo(42);
}

Same error:

main.d(4): Error: namespace `main.namespace` conflicts with namespace main.namespace at main.d(3)

So the right way to write the function declarations would be:

module main;

extern(C++, namespace) {
    void foo(int);
    void bar(const char*);
}

void main ()
{
    bar("Hello world".ptr);
    foo(42);
    namespace.bar("Hello world".ptr);
    namespace.foo(42);
}

Wait, what ? I can refer to bar and foo using both the namespace qualifier and without it from outside the namespace ? Even C++ does not allows that without using namespace (or inline namespace).
And no other place in D allow that either, a scope is only ever imported into the current scope using explicitly, using alias.
Well, there's one place where a symbol was available both qualified and unqualified: import. But we fixed that bug a while ago.

Oh well, whatever, I'll just bind std::vector and go on my merry way:

--- a.d
module a;
extern(C++, std) struct vector(T) { T*[3] data; }
--- main.d
module main;

import a;
import std.stdio;

void main ()
{
    std.vector!int vec;
    writeln("Pointers: ", vec.data[0], ", ", vec.data[1], ", ", vec.data[2]);
}

main.d(9): Error: template identifier vector is not a member of import main.std

What happened here ? Well imported packages (in this case std of std.stdio) create a private symbol in the local scope, which takes precedence over imported symbol... Oops. But "luckily" using vector!int instead of std.vector!int will work so that's what I'm going to have to do if I ever want to use C++ interop and the standard library.

Wait... There's more!
Now if I define my binding in my local scope, I have a bit of a problem:

module main;
import std.stdio;
extern(C++, std)       struct vector(T) { T*[3] data; }

This results in the following error:

main.d(3): Error: namespace `main.std` conflicts with import main.std at main.d(3)

Because -of course- I am trying to define a symbol which has the same name as a package I import.
This would happen as well with the following code:

module main;
import std.stdio;
struct std {}

But the key difference here is that the struct definition is under the developer's control while the namespace name is an externally defined identifier on which the developer might have no control over but has to import in his/her code. The only solution here is for me to change all my std.* imports to renamed imports. This of course also happens for function names (with extern(C)) but those tends to be much more specific and carefully picked than namespace names.

TL;DR:
Introducing a scope for the namespace messes with splitting the definition of the namespace over multiple modules, which C++ does all over the place and is something we absolutely need. Just look at the standard library. Symbols are being injected in two scopes implicitly, which is unprincipled. Qualified version of standard types (std.symbol) will be unusable in non-trivial program so people will have to stick to the unqualified version.

Now go back to all the examples, one by one. Remove the qualification at call site (so namespace.foo becomes just foo) and use extern(C++, "namespace"). Everything will just work.

[TurkeyMan]

^^ All of those words. It's a huge mine field, and no D programmer wants added complexity like that.
When you have to rename virtually every import to workaround collisions, then the feature self-defeats.

[WalterBright]

An objective fact no one but you seem to agree on.

I hazard a guess that I'm the only one in this discussion who has read the C++ Standard section on namespaces. Section 3.3.5 Namespace scope and Section 7.3 Namespaces of the C++98 spec. I suggest reading them before telling me C++ namespaces don't have a scope.

Any idea when you will have a clearer picture ?

The example I gave earlier makes it pretty clear. It works in C++, it fails in D and D code (the string version) does not compile and cannot connect to the C++ version. Feel free to try it yourself.

[WalterBright]

#include <stdio.h>
namespace A { int foo() { return 1; } }
namespace B { int foo() { return 2; } }
void main() {
    printf("%d %d\n", A::foo(), B::foo());
}

Prints 1 2. The D version using strings doesn't compile.

[SSoulaimane]

The example I gave earlier makes it pretty clear. It works in C++, it fails in D and D code (the string version) does not compile and cannot connect to the C++ version. Feel free to try it yourself.

@Geod24 gave the counter example, which shows why half way implemented features don't work. You are complaining that the new syntax doesn't match C++ namespaces, and we are complaining that neither the old syntax does. At least the for the new syntax people know what to expect, they have to use D modules for grouping symbols.

[SSoulaimane]

...
Prints 1 2. The D version using strings doesn't compile.

Because it's not D. Use modules for grouping in D not namespaces.

[WalterBright]

At least the for the new syntax people know what to expect, they have to use D modules for grouping symbols.

How are you going to do nested namespaces?

Manu didn't want to reorganize his code to group all declarations in one namespace together, and didn't want to use alias to merge scopes. But you're suggesting that each namespace go in its own file.

I suggested using alias in the n.g. debate on this to Manu, multiple times. It works fine. @Geod24 did not mention this in his rebuttal above.

I'm also not suggesting removal of the string namespaces. I gave up on that, even though I consider it wretched and would never use it myself (using alias instead). What I reject is trying to delete scoped namespaces from D, this would be a serious error.

[WalterBright]

@Geod24 I can refer to bar and foo using both the namespace qualifier and without it from outside the namespace ? Even C++ does not allows that without using namespace (or inline namespace).
And no other place in D allow that either, a scope is only ever imported into the current scope using explicitly, using alias.

This is incorrect. Both imports and Template mixins do the same thing. It's a nice convenience feature, and works as long as there are no ambiguities.

[TurkeyMan]

#include <stdio.h>
namespace A { int foo() { return 1; } }
namespace B { int foo() { return 2; } }
void main() {
    printf("%d %d\n", A::foo(), B::foo());
}

Prints 1 2. The D version using strings doesn't compile.

I've never seen code like that in my life. Nobody does that.
namespaces are a poor-mans substitute for modules, except in reality, they aren't used to represent modules, they are used to represent packages... entire packages, which span many files across a projects source hierarchy.
You know this is trivial to resolve using D modules, and if you're a D programmer writing D code, that's the solution you should use. There is no desire to have C++ namespace semantics in D, nobody has ever asked for that.
Your not-real-world example does not negate @Geod24's entire post above, which represent real-world issues.

How are you going to do nested namespaces?

How does any D program do it? With nested modules, obviously.

Manu didn't want to reorganize his code to group all declarations in one namespace together, and didn't want to use alias to merge scopes.

Code is distributed among many headers which gather code in a module-like fashion. No I do NOT want to take 100 header files and collapse them into one module, that's a ridiculous proposition!
It's completely natural and expected to distribute the headers among modules and mirror the organisation. Literally nobody would expect anything different than that, and they would be angry if you did something else.

But you're suggesting that each namespace go in its own file.

This is how D works. We are writing D code. The explicit goal here is to NOT write C++ code.
Maybe if D had a named scope feature, we might take advantage of that on occasion. But it doesn't.

I suggested using alias in the n.g. debate on this to Manu, multiple times. It works fine. @Geod24 did not mention this in his rebuttal above.

It's pointless busy work, and it doesn't eliminate the problems, it only mitigates the situation a little bit.

Aliasing symbols out of their namespace into the module scope is an admission of defeat, so you've already lost and just forcing boilerplate and busy-work on us at this point... but it also doesn't mean the namespace symbols are not there in the module scope. Import a phobos module, the namespace is still present and it will conflict.

There's also still the problem where valid C++ identifiers and valid D identifiers are different sets. We have C++ namespaces with invalid D identifiers. C++ namespaces can't have a D identifier, otherwise you just have a ticking bomb until someone hits a name outside the intersection.

This is incorrect. Both imports and Template mixins do the same thing. It's a nice convenience feature, and works as long as there are no ambiguities.

There ARE ambiguities though; STL and phobos, which have a tendency to coexist in every file that has extern(C++) in it.

What I reject is trying to delete scoped namespaces from D, this would be a serious error.

Personally speaking, I'm not trying to coerce you to do this. I don't care if your scoped namespaces remain; but I do think that if you want a named scope, make a named scope feature. That would be the cool thing to do.

[TurkeyMan]

Also, this is the most uninteresting use of your time imaginable (and ours too).
I don't know why you get so triggered by this issue that literally doesn't affect you at all. There are so many things that really need your attention, and this is just not one of them.

[WalterBright]

BTW, C++ indeed does allow reopening a namespace and inserting more members. The reason is that C++ doesn't do forward references. C++ structs do allow forward referenced members, and doesn't allow reopening the struct.

[WalterBright]

I've never seen code like that in my life. Nobody does that.

There are a million C++ programmers. You haven't seen a remote percentage of the code they write. I invite you to submit a proposal to the C++ Standards Committee to remove namespace scopes and make it mangle-only. You'll find out real quick who relies on it.

I do NOT want to take 100 header files and collapse them into one module, that's a ridiculous proposition!

Import the 100 header files into one module, and add 100 aliases to that module, and you're done. And you've got a principled solution.

There's also still the problem where valid C++ identifiers and valid D identifiers are different sets.

That's a general issue, not specific to namespaces, and is not a scoping thing.

I don't know why you get so triggered by this issue that literally doesn't affect you at all.

I get triggered when features get deleted due to resolute misunderstanding.

[TurkeyMan]

There are a million C++ programmers. You haven't seen a remote percentage of the code they write.

And how many of them are here investing years of their life trying to be successful with D?
Understand, all I want to do is vindicate decades of your life's work! That doesn't mean I have to agree you got everything right long the way, but we're really REALLY trying here!!

What we have is a small sample of REAL ACTUAL USERS, they are all in agreement here, and couldn't be clearer about the tool we need. Focus on the people that matter, and if someone comes along in the future and says "I want a namespace scope", talk to them and hear their story...

Don't disregard the people standing right here while saying some bullshit about a million people that don't give a damn.

I get triggered when features get deleted due to resolute misunderstanding.

I'm not deleting it, I'll leave that for others to argue about. It doesn't affect me, but I am disappointed that it's not just a bonafide named scope in its own right, that might be generally useful on occasion..

[WalterBright]

This works:

module a; extern (C++, ns) { int foo() { return 1; } }

module b; extern (C++, ns) { int bar() { return 2; } }

import a, b;
static assert(foo() == 1 && bar() == 2);

[WalterBright]

This also works:

module a; extern (C++, ns) { int foo() { return 1; } }

module b; extern (C++, ns) { int bar() { return 2; } }

module ns;
import a, b;
alias foo = a.ns.foo;
alias bar = b.ns.bar;

import ns;
static assert(ns.foo() == 1 && ns.bar() == 2);

It's no worse than writing headers in C++.

[Geod24]

This works: [...]

Because you are not using their qualified name... As @TurkeyMan said:

Aliasing symbols out of their namespace into the module scope is an admission of defeat, so you've already lost and just forcing boilerplate and busy-work on us at this point...

@WalterBright said:
This also works: [...]

It does. And so does its string equivalent:

--- a.d
module a; extern (C++, `ns`) { int foo() { return 1; } }
--- b.d
module b; extern (C++, `ns`) { int bar() { return 2; } }
--- ns.d
module ns;
public import a, b;
--- main.d
import ns;
static assert(ns.foo() == 1 && ns.bar() == 2);

So what's the point of introducing a scope just to remove it because it always conflict ?

It's no worse than writing headers in C++.

C++ is not the baseline here, D is.

[jacob-carlborg]

FYI, there are a lot of things where D doesn't match the other language in a language integration. Examples:

• We don't support multiple inheritance of extern (C++) classes in D
• We do support final extern (C++) and extern (Objective-C) methods. It doesn't exist in C++98 and it doesn't exist in Objective-C
• extern (Objective-C) methods support overloading while Objective-C doesn't

Just to mention a few.