add __totype(string) expressions

[WalterBright]

This adds a new Type construct:

__totype("i") x;

where __totype takes as its argument a string that is a mangled type, and produces the corresponding type (in this case int), which can be used where-ever a Type is used.

This, in effect, closes the circle where .mangleof produces a string literal representing the type, and __typeof correspondingly converts the string back into a type. It's an obvious (in retrospect) building block in D's metaprogramming.

What's it good for? Quite simply, it enables a type to be treated as an expression, along with everything an expression can do. Being a compile-time construct, the expression handed to it is fully evaluated by CTFE. It has the potential to completely obsolete typeid and all the compiler magic that comes with it, i.e. a major simplification and a big boost in metaprogramming power.

This is a prototype for experimenting and for proof of concept. It's kinda shocking how little code is required for it.

[dlang-bot]

Thanks for your pull request, @WalterBright!

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub run digger -- build "master + dmd#11797"

[UplinkCoder]

This is a polymorphic (shape changing) expression.

If you used this in a ctfe function, and declared a variable with the resulting type;
you change the stack frame size, which is what I call a shape change.

The shape of the memory or instruction layout changes.

[ibuclaw]

Is there a bugzilla, dip, or any other reference available to track where this request came from?

[UplinkCoder]

This is introducing a polymorphic/ add a test which uses __totype within a function and depends on CTFE (runtime) argument, and you'll see it doesn't work.
(for the same reason that a string mixin depending on a runtime argument doesn't work)

It would be somewhat useless if wanting to remove dependence on recursive templates is the goal.

[andralex]

It has the potential to completely obsolete typeid and all the compiler magic that comes with it, i.e. a major simplification and a big boost in metaprogramming power.

So... does that mean my beloved typeid is not salvageable? :o|

This is definitely something powerful for a budget of 172 lines. I have a question: in an AliasSeq one could put types, symbols, or expressions - e.g. AliasSeq!(int, xyz, 1+1). With mangleof and __totype we get the full circle for types. How about symbols and expressions though? I noticed that .stringof for symbols and expressions return that thing as a string, but I'm not sure it works in all cases (e.g. taking a symbol to another scope). What would it take to close the circle with symbols and expressions as well? Thanks.

[andralex]

This is a fix for https://issues.dlang.org/show_bug.cgi?id=9945 it seems.

[quickfur]

This is a polymorphic (shape changing) expression.

If you used this in a ctfe function, and declared a variable with the resulting type;
you change the stack frame size, which is what I call a shape change.

The shape of the memory or instruction layout changes.

Not really. The argument to __totype must be known at compile-time, that is, at template expansion time, so it cannot be a variable within the current function, but must be a fixed compile-time value. I.e., you cannot do this:

auto ctfeFunc(Args...) {
    string x = /* some string expression */;
    return __totype(x).init; // compile error: x cannot be read at [AST] compile-time
}

Instead, you have to do this:

string ctfeFunc(Args...) {
    string x = /* some complicated expression */;
    return __totype(x); // compile error: x cannot be read at [AST] compile-time
}
alias T = __totype(ctfeFunc(...)); // OK

(@UplinkCoder wanted to quote and accidentally edited the comment)
@quickfur please re-add.

[andralex]

If you used this in a ctfe function, and declared a variable with the resulting type; you change the stack frame size, which is what I call a shape change.

That would be a real code generation bug that probably affects mixin code as well. Can you please give an example? Thanks.

[adamdruppe]

i wanna follow this. it is basically just mixin that skips the import scopes but that's legit useful i think. but slight concern on dependency tracking and linker errors though; it might work in dmd, but not right with make and ld (well dmd -i really) and such exactly because those are more likely to follow the import path and thus not realize these dependencies necessarily even exist.

[UplinkCoder]

This is a polymorphic (shape changing) expression.
If you used this in a ctfe function, and declared a variable with the resulting type;
you change the stack frame size, which is what I call a shape change.
The shape of the memory or instruction layout changes.

Not really. The argument to __totype must be known at compile-time, that is, at template expansion time, so it cannot be a variable within the current function, but must be a fixed compile-time value. I.e., you cannot do this:

auto ctfeFunc(Args...) {
    string x = /* some string expression */;
    return __totype(x).init; // compile error: x cannot be read at [AST] compile-time
}

Instead, you have to do this:

string ctfeFunc(Args...) {
    string x = /* some complicated expression */;
    return __totype(x); // compile error: x cannot be read at [AST] compile-time
}
alias T = __totype(ctfeFunc(...)); // OK

This is exactly what polymorphic is.
polymorphism in this case means => it has to happen at "template expansion time"

[UplinkCoder]

If you used this in a ctfe function, and declared a variable with the resulting type; you change the stack frame size, which is what I call a shape change.

That would be a real code generation bug that probably affects mixin code as well. Can you please give an example? Thanks.

I can't give an example because it doesn't work.
I won't compile if you try that.

The compiler has semantics which include an invariant that ensures functions are mono-morphic.

[WalterBright]

You're right, the string can't be used as a runtime argument to a function. But it can be used as a compile-time argument:

int foo(string s1, string s2)()
{
    __totype(s1) x = 4;
    mixin(s2) y = 3;
    return x + y;
}

enum i = foo!("i", "int")();

just like mixins.

[UplinkCoder]

We will see how far you get with that.

It's a step.

What change is requested?

[WalterBright]

macOS 10.15 x64 heisenbug is a constant annoyance.

[WalterBright]

macOS 10.15 passing now, proving it was another heisenbug, as no changes were made.

[AndrejMitrovic]

and __typeof correspondingly converts the string back into a type

did you mean __totype?

[WalterBright]

did you mean __totype?

Yes.

[WalterBright]

Now an unrelated buildkite failure:

Running the testsuite | 0s
 | --
  | ~> case "$REPO_FULL_NAME" in
  | ~> export TRAVIS_OS_NAME=none
  | ~> TRAVIS_OS_NAME=none
  | ~> use_travis_test_script
  | ~> echo 'set -xeu'
  | ~> /var/lib/buildkite-agent/builds/ci-agent-6226e41f-df0b-4254-85f7-af720b10947d-2/dlang/dmd/build/buildkite/travis_get_script
  | ~> bash
  | ~> sed -e /meson/d
  | + [[ '' == \l\d\c\2 ]]
  | bash: line 2: TRAVIS_CPU_ARCH: unbound variable
  | 🚨 Error: The command exited with status 1

[rainers]

I think this is way too simplistic: you cannot synthesize new types and you cannot create types that the compiler hasn't yet seen, but are in other modules, structs or similar.

I don't yet see how __totype does anything more than optimize mixin(core.demangle(deco)) (but allowing access to private types).

[andralex]

Ah, so that only works with strings that were previously returned by .mangleof. Yes that may be problematic.

[andralex]

All in all this is nice and dandy but I have one objection and one suggestion:

• Whenever we introduced similar facilities in D, we used __traits. This stands out, particularly because it's definitely not to be used naively so verbosity is a nonissue. I suggest packaging the feature as __traits(mangleToType, str).

• The facility should accept multiple items and return a built-in tuple of types: __traits(mangleToType, "i", "i") should yield AliasSeq!(int, int).

I'll also note we don't have a solution to converting (reliably) non-type aliases to strings and back.

[adamdruppe]

On Sun, Sep 27, 2020 at 04:47:15PM -0700, Andrei Alexandrescu wrote: * The facility should accept multiple items and return a built-in tuple of types: `__traits(mangleToType, "i", "i")` should yield `AliasSeq!(int, int)`.

That is excellent, but you know what would be really top level? If it could take a CTFE'd array of strings directly.

…

__traits(mangleToType, ["i", "i"]) would return AliasSeq!(int, int) exactly the same as __traits(mangleToType, "i", "i"). (D's type-safe variadics work this same way btw). That would significantly simplify using this in practice. Cut out another helper template AND you can just return the value from CTFE and use it directly without a helper mixin as well!

[adamdruppe]

BTW a general facility of array (obviously at least length known at compile time, but contents may or may not be known there; ideally, ONLY the length would be CT required, so it can be used on runtime params too) to tuple would also be really useful. Doing that in the library is a mess with random limitations, but the compiler could make it easy.

Manu's ... static map proposal of course does that and more - so I'm still in favor of that - but if we just had some facility for array to tuple then you wouldn't have to handle that inside this specific trait.

[BorisCarvajal]

But why not just create a demangle property for strings and use mixins?
mixin("i".demangle) var;
That should work with symbols too.

You can mark StringExp as a decoded type to reduce any overhead in the mixin logic.

[andralex]

But why not just create a demangle property for strings and use mixins?
mixin("i".demangle) var;
That should work with symbols too.

You can mark StringExp as a decoded type to reduce any overhead in the mixin logic.

Hm, I thought this doesn't work, but it does:

mixin("int") x;

[thewilsonator]

Hm, I thought this doesn't work,

It used to not. It was changed a couple of years ago.

[adamdruppe]

On Mon, Sep 28, 2020 at 02:22:09AM -0700, Boris Carvajal wrote: But why not just create a `demangle` property for strings and use mixins? `mixin("i".demangle) var;`

The problem with this is scope lookup. Suppose you had like _D4test5thing which is easy enough to demangle: test.thing. (And I think there's a letter in there to indicate it is a struct but that's beside the point rn). mixin("test.thing") has two major problems: 1) If you haven't done `import test;`, the name is out of scope. Remember, you can pass aliases to templates in different scopes and the mixin will be evaluated there, not at the original point. So, for example, you pass that to std.algorithm, and since std.algorithm hasn't `import test;`, it will get "undefined identifier: test". 2) Worse yet, related to the same issue, it is possible for it to refer to something *different*. Suppose you had two modules: ``` module test; struct thing {} import cool; cool.identity!thing t; ``` And ``` module cool; template identity(T) { private struct test { static struct thing {} } alias identity = mixin(T.mangleof.demangle); } ``` The demangled string there - test.thing - in this different context is now giving an entirely different entity! Your dereified data is corrupted. (The mangle of that inner thing would be like `cool.identity!T.test.thing`, it wouldn't match the original one... but mixin doesn't know that, it just sees the demangled name.) So existing mixin won't work. This is the main thing I always come back to when I tell people "NEVER use .stringof" - it always either doesn't work, works incorrectly, or at the very least, even when it works, is the ugliest thing that works (you'd be better off just using `T` - like my example on the forum, using indexes into the local Args array - which I do think is a serious competitor to this new feature. It might be good enough to do it all as-is, just a bit awkward, with the local index pattern.) * * * This PR *basically* is mixin, so we can reason about it by analogy in terms of what we know, but it does have this very important key difference in that it skips language scoping rules. So this particular problem is bypassed.

[adamdruppe]

It used to not. It was changed a couple of years ago.

that's cool. btw if void mixin(name)() {} worked you'd cut my number of larger string mixins in like half instantly. just fyi :)

[andralex]

I just realized that this feature is not synonymous to mixin("int") because it operates on the mangled name, not the clear string.

Different topic - here are a few __traits that are akin to __totype. These strengthen the case on why we should package this feature as a trait:

• __traits(getAliasThis, T) yields a type
• __traits(getMember, s, "T") where T is a member type of typeof(s) yields a type
• Similarly with parent and child

[TurkeyMan]

This, in effect, closes the circle where .mangleof produces a string literal representing the type

This wouldn't necessarily be true if I manage any traction with this: https://issues.dlang.org/show_bug.cgi?id=21203
Which hasn't received any attention... and I think it's critical to close the outstanding issues with C++ interop.

[BorisCarvajal]

This PR basically is mixin, so we can reason about it by analogy in terms of what we know, but it does have this very important key difference in that it skips language scoping rules. So this particular problem is bypassed.

@adamdruppe That could be implemented as another feature, something that let you use/create types (or forward references of them) from strings, not necessarily from a mangled one.

[UplinkCoder]

@BorisCarvajal creation of types is quite problematic, because they need a mangle.
If you use a function to create a type it needs to be strongly pure, and even then you to have access to where-ever it was created from.
Which is the reason why I omitted this feature from type functions.

[UplinkCoder]

@WalterBright

we should package this feature as a trait

While I seldom agree with his views, Andrei is right about this one.

If we do it at all.

[12345swordy]

ping

[WalterBright]

I think I agree with @andralex that this should be re-implemented as a __trait. It'll be drastically simpler.

[WalterBright]

https://issues.dlang.org/show_bug.cgi?id=21585