Capability to get children compile-time info. #775

yglukhov · 2014-04-23T15:18:54Z

This commit adds a feature of compile-time introspection of aggregate types' "children".
E.g.

struct MyCustomInfo(T)
{
    static this()
    {
        writeln("Registering runtime info for ", T.stringof);
    }
}

@rtInfo!MyCustomInfo class RootOfSomeRuntimeRegisteringHierarchy
{
}

class A : RootOfSomeRuntimeRegisteringHierarchy
{
}

Will output:

Registering runtime info for RootOfSomeRuntimeRegisteringHierarchy
Registering runtime info for A

This will need some documentation if going to be merged. I'll write it as soon as all other questions are settled.

MartinNowak · 2014-04-23T19:51:17Z

src/object.di

+        foreach(i; __traits(allMembers, T))
+            static if (i == "_RTInfo")
+                return true;
+        return false;


Could you do that without the looping? I'm a bit worried about the compile time impact because _RTInfo is instantiated for every type.

Unfortunately, i could not get it right without the loop. The main reason for the loop is not to be tricked with opDispatch. I think that such things (_RTInfo) should be explicitly defined and not with opDispatch, otherwise strange things could implicitly happen. This is arguable though.

As to performance, i didn't notice any difference on my core i7 2.9Ghz/16GB ram macbook, but of course i didn't make any scientific research on that, sorry.

What about __traits hasMember? Agreed upon the opDispatch thing.

__traits hasMember is fooled miserably by opDispatch =(
Even worse, if using __traits hasMember as a precondition before the loop, it will instantiate this template. This is a tricky use-case, i stumbled upon in my objective-c bridge. I can provide more info on that, if needed.

MartinNowak · 2014-04-23T19:52:47Z

I like the idea very much. A small newsgroup discussion would be nice.

MartinNowak · 2014-04-23T19:54:36Z

src/object.di

@@ -577,6 +577,15 @@ void __ctfeWriteln(T...)(auto ref T values) { __ctfeWrite(values, "\n"); }

 template RTInfo(T)
 {
+    static if ({


Please also update object_.d.

Done. However, i don't quite understand why it is needed in object_.d.

Because object.di is just the header file to object_.d, but as we're currently not able to generate the header automatically we need to maintain both. Another reason is that unit tests only run on object_.d.

In such case, why does object_.d declare RTInfo(T).RTInfo as

enum RTInfo = null;

while object.di declares it as

enum RTInfo = cast(void*)0x12345678;

?

yglukhov · 2014-04-23T20:11:58Z

Well, i've started a humble thread here, though it has not drawn too much attention yet.
http://forum.dlang.org/thread/bqyazxxdkpruxlxwfhmi@forum.dlang.org

MartinNowak · 2014-04-23T22:33:02Z

Static this registering is nice already, but I wonder if it's also possible to inject this into TypeInfo somehow.

jacob-carlborg · 2014-04-24T06:08:42Z

I like the idea very much. A small newsgroup discussion would be nice.

I don't like this solution, see newsgroup discussion.

MartinNowak · 2014-04-24T09:30:13Z

Can you work with @jacob-carlborg to use a UDA attribute on the type instead of a nested template? Then this might also work for enums.

yglukhov · 2014-04-24T13:19:24Z

Ok, i've played a bit with UDA, and here is what i've got now.

struct MyCustomInfo(T) // or alias T
{
    static this()
    {
        writeln("Registering runtime info for ", T.stringof);
    }
}

@rtInfo!MyCustomInfo class RootOfSomeRuntimeRegisteringHierarchy
{
}

class A : RootOfSomeRuntimeRegisteringHierarchy
{
}

Will output:

Registering runtime info for RootOfSomeRuntimeRegisteringHierarchy
Registering runtime info for A

What do you think?

jacob-carlborg · 2014-04-24T17:18:36Z

I was thinking something like this:

@rtInfo template MyCustomInfo (T)
{
    // inspect T
}

I want to be able to inspect third party types I don't have any control over.

MartinNowak · 2014-04-24T22:24:05Z

I want to be able to inspect third party types I don't have any control over.

As I've said already, we cannot inject code into foreign modules, because it conflicts with separate compilation. What you can do is import foreign modules and inspect the hell out of them.

MartinNowak · 2014-04-24T22:28:31Z

src/object.di

 template RTInfo(T)
 {
+    template _instanciateRTInfoFromAttributes(Attrs...)


It's "instantiate" :).

yglukhov · 2014-04-24T22:32:31Z

Actually, I'm currently trying to develop something that might suite all of us. What if we mix the two approaches of Jacob and mine in the following way:

A module can define @rtInfo template MyCustomInfo (T), as Jacob says.
All such templates will be instantiated on everything the module defines, as well as it's "children", that is the modules that include it.

This could possibly work, and cover much more use-cases than just children CT-info, and still remain consistent. There's still a small question regarding cyclic imports, but i guess, ill have an answer soon.

MartinNowak · 2014-04-24T22:34:03Z

src/object.di

+    {
+        static if (Attrs.length == 0)
+            alias _instanciateRTInfoFromAttributes = _Tuple!();
+        else static if (is(Attrs[0] == rtInfo!(U), alias U))


This should work with multiple rtInfos as in @rtInfo!(Reg1, Reg2).

MartinNowak · 2014-04-24T23:02:03Z

Actually, I'm currently trying to develop something that might suite all of us.

OK, I'm curious.

yglukhov · 2014-04-25T14:16:38Z

Ok, i've got some results, but a bit shy to show them, and that's probably not the right way to go. Here's what i've done:

Expanded __traits(allMembers, T) to return imports of a module, if T is module.
Expanded RTInfo(T) to get T's module, and recursively get all the modules, imported by T's module.
Every such module is searched for templates marked with @rtInfo.
Found templates are instantiated on T.

The whole thing works, with a few buts.

You'll need at least 12GB available ram to build something like phobos.
You'll need at least 5 minutes for that also, instead of 3 seconds =).
The built binary of phobos lib takes about 110MB, instead of 16MB. Seems like i'm over-instantiating everything.
Also i had to increase 500-nest threshold in DMD.

The way to go, IMHO:

Do everything almost like Jacob suggests, but instantiate the @rtInfo-marked templates only on the members of currently-compiled module, and collect @rtInfo-marked templates only in the currently-compiled module and the modules it imports.

In other words, do everything i've done experimenting, but with compiler's help.

This way a module can control it's "children" just like a superclass can control it's subclasses in my PR here. There will be no link-time ambiguity.

If a module wants to generate something for the modules it imports, it already may do so with existing language features.

I still have nothing to say about rtInfo pointer, maybe that's because i still can't fully understand the reason to have any "default" runtime at all, except for cpp-ish dynamic_casts.

So what do you think guys?

MartinNowak · 2014-04-25T15:31:56Z

Do everything almost like Jacob suggests, but instantiate the @rtInfo-marked templates only on the members of currently-compiled module, and collect @rtInfo-marked templates only in the currently-compiled module and the modules it imports.

Yeah, this is roughly what we talked about in dlang/dmd#2271. I'm still a bit worried about the performance implications of such a feature.
Note that you can cache the list of "active" @rtInfo template in a certain module by instantiating a template like so.

alias getRTInfoTemplates(alias mod) = TypeTuple!(
    filterRTInfoTemplates!(__traits(allMembers, mod)),
    staticMap!(getRTInfoTemplates, importedModules!mod))
);

The compiler will cache templates with identical arguments and can find them in O(1) (hashtable lookup).

Now you'd only need to instatiate all of the relevant templates in rtInfo.

alias instantiateTemplates(T) = TypeTuple!();
alias instantiateTemplates(T, Tmpls...) = TypeTuple!(staticMap!(Tmpls[0], T), instantiateTemplates!(T, Tmpls[1 .. $]));
alias rtInfo(T) = instantiateTemplates!(T, getRTInfoTemplates!(moduleOf!T));

jacob-carlborg · 2014-04-26T09:30:17Z

Do everything almost like Jacob suggests, but instantiate the @rtInfo-marked templates only on the members of currently-compiled module, and collect @rtInfo-marked templates only in the currently-compiled module and the modules it imports.

Why this limitation? It also makes it more complicated, at least do document and explain. Why not just instantiate all @rtInfo templates with all user defined types the compiler sees. Simple to explain, simple to understand.

Basically the only case your approach doesn't work for is when passing at least two files two the compiler and they don't import each other.

Another idea is to skip RTInfo and implement RMInfo instead. I think it should be possible for RMInfo to inspect all user defined types in all modules.

[

yglukhov · 2014-04-26T16:18:10Z

Simple to explain, simple to understand.

Well that depends on how you put it. If we're talking about a language that is compiled and linked, your way would not work consistently if we're compiling/linking partially. Second there's dependencies. Now we know that every module B that imports module A depends on module A, meaning that changing A would require rebuilding B, otherwise linking with "outdated" B object leads to undefined result.

Also there is semantic reason. Instantiating customs on everything leads to totally uncontrolled environment. All of your subsystems automatically become dependent on everything you link with, and you'll have a hard time tracking who does custom RTInfo over your own code.

Even with my way_to_go the responsibility of writing an @rtInfo marked template is enormous. This feature is extremely powerful, and should be used with great care. I clearly imagine that this feature will be banned in commercial projects on D, just like excessive use of preprocessor or Alexandrescu-style templates in cpp.

jacob-carlborg · 2014-04-27T08:58:42Z

Now we know that every module B that imports module A depends on module A, meaning that changing A would require rebuilding B, otherwise linking with "outdated" B object leads to undefined result.

I'm not sure that's the case. As long as the module containing the @rtInfo template is processed by the compiler, one way or another it should be enough.

But I guess explicitly importing the module might be safer and not that big of limitation that I first thought.

Also there is semantic reason. Instantiating customs on everything leads to totally uncontrolled environment. All of your subsystems automatically become dependent on everything you link with, and you'll have a hard time tracking who does custom RTInfo over your own code.

We already have that with the current RTInfo.

So if I understand you correctly: Instantiate all @rtInfo templates that are imported, one way or another, with all user defined types seen by the compiler? Or do I have to explicitly import a module for it to include those types?

What should we do about the current object.RTInfo? Leave it as is, remove it or add the @rtInfo UDA?

BTW, we do need a way to instantiate a template with all types (not just user defined types) seen by the compiler to be able to build a precise garbage collector. This is the reason RTInfo was created in the first place.

yglukhov · 2014-04-27T11:06:40Z

We already have that with the current RTInfo.

No we don't, since RTInfo cannot be custom now.

So if I understand you correctly: Instantiate all @rtInfo templates that are imported, one way or another, with all user defined types seen by the compiler? Or do I have to explicitly import a module for it to include those types?

I would say this way: Instantiate all @rtInfo templates that are imported, one way or another, with all types defined in currently-compiled module.

What should we do about the current object.RTInfo? Leave it as is, remove it or add the @rtInfo UDA?

I'm currently trying to use it as an entry point for @rtInfo customs instantiations, but i'm not really successful in it. I followed @MartinNowak's advise to cache per-modules infos and it worked, and now we need just 50 seconds to build phobos, and 8GB ram would be enough, but i need somehow to crank it down to normal values, and also i could not solve the final binary size issue. So returning to RTInfo, i think it should be either as it is used now - the root for all custom rtinfos, or it may be removed completely, if i do the job with compiler's help, or somewhere in the middle.

BTW, we do need a way to instantiate a template with all types (not just user defined types) seen by the compiler to be able to build a precise garbage collector. This is the reason RTInfo was created in the first place.

And such possibility will be available, if GC's module that defines it's @rtInfo will be included somewhere in the core.

jacob-carlborg · 2014-04-27T11:19:12Z

I would say this way: Instantiate all @rtInfo templates that are imported, one way or another, with all types defined in currently-compiled module.

So that means if I only compile module A which imports module B, only the types in A will be instantiate, not the ones in B?

And such possibility will be available, if GC's module that defines it's @rtInfo will be included somewhere in the core.

That doesn't sound possible with what you just said above. Or I'm misunderstanding.

rainers · 2014-04-28T07:27:23Z

Please note that RTInfo isn't generated reliably at the moment. I've been trying to get fixes merged for ages: dlang/dmd#2480

yglukhov · 2014-04-28T08:17:34Z

So that means if I only compile module A which imports module B, only the types in A will be instantiate, not the ones in B?

That's right. Types in B will be instantiated when compiling B. Isn't it the way current RTInfo works?

That doesn't sound possible with what you just said above. Or I'm misunderstanding.

If you'll have and @rtInfo imported somewhere on early stages (core?), it will be instantiated pretty much on everything you compile.

rainers · 2014-04-28T08:28:02Z

That's right. Types in B will be instantiated when compiling B. Isn't it the way current RTInfo works?

Almost, but not completely. I tried to summarize it here: https://github.com/D-Programming-Language/dmd/pull/2480/files#diff-3344a3f04733c74c9601b00e0b399ca0R10

yglukhov · 2014-04-28T09:21:45Z

@rainers, so do you think it's worth waiting for your PR to be merged, before continuing on custom rtInfos? Do you know, when it will be merged btw? =) Or do you think we should merge my current PR meanwhile?

yglukhov · 2014-04-28T11:23:19Z

Just in case anyone interested in my progress on custom @rtInfos, here's the code with all the problems described above:
https://github.com/yglukhov/druntime/blob/uglyRTInfoExperiment/src/object.di
This code depends on my patch to DMD, that enables __traits(allMembers) to return a list of imports, which is also a bit ugly, as they are returned as fully-qualified names of imports. E.g. Tuple!("std.stdio", "std.conv").

jacob-carlborg · 2014-04-28T13:26:44Z

That's right. Types in B will be instantiated when compiling B. Isn't it the way current RTInfo works?

Hmm, I think I need to actually check how RTinfo works.

If you'll have and @rtInfo imported somewhere on early stages (core?), it will be instantiated pretty much on everything you compile.

There might be some confusion here. When I say "compiled", I'm referring to the file you actually pass to the compiler on the command line, not a file which is imported.

rainers · 2014-04-28T18:10:51Z

@rainers, so do you think it's worth waiting for your PR to be merged, before continuing on custom rtInfos?

As long as you don't hit problems with missing RTInfo, it will probably not affect your implemetation.

Do you know, when it will be merged btw? =)

Probably not as long as Walter or Kenji hit the same problems ;-( (Kenji did a bit in the same direction fixing opCmp/opEqual members in the type info) I stopped pleading for review some time ago.

Or do you think we should merge my current PR meanwhile?

I'm a little uneasy about it. Some questions:

How does it affect compilation speed?
Do symbol lookup rules allow anything useful? What about private members?
Could you add some (unit)tests?

yglukhov · 2014-04-30T06:11:46Z

How does it affect compilation speed?

I'm not sure how to test that properly. Building phobos on my macbook 2.9GHz, 16GB ram:
Without my changes:

$ time make -f posix.mak
real    0m4.056s
user    0m3.695s
sys 0m0.339s
$ time make -f posix.mak
real    0m4.095s
user    0m3.735s
sys 0m0.331s

With my changes:

$ time make -f posix.mak
real    0m4.230s
user    0m3.832s
sys 0m0.351s
$ time make -f posix.mak
real    0m4.304s
user    0m3.909s
sys 0m0.372s

Do symbol lookup rules allow anything useful? What about private members?

Sorry, didn't understand that. I'm not looking up for members anywhere. Could you clarify your question pls?

Could you add some (unit)tests?

Of course i will, but as far as i understand, the tests on this feature should be added to DMD, not runtime, right?

MartinNowak · 2014-05-16T17:32:03Z

Of course i will, but as far as i understand, the tests on this feature should be added to DMD, not runtime, right?

Tests for instantiating rtInfo belong to dmd, tests for rtInfo itself belong to druntime.

yglukhov · 2014-06-04T18:30:52Z

Hm, can't find a good spot to insert the unittest. When i add a test to object_.d, it would not link, saying the following:

Undefined symbols for architecture x86_64:
  "_D4core3sys3osx7pthread12__ModuleInfoZ", referenced from:
      _D4core6thread12__ModuleInfoZ in test_runner.o

static this is what's causing link error. Here's the test i'm trying to add:

version(unittest)
{
    static string[] registeredClasses;
    struct CustomRTInfo(T)
    {
        static assert(is(T == A) || is(T == B));
        static this()
        {
            registeredClasses ~= T.stringof;
        }
    }

    @rtInfo!CustomRTInfo class A { }
    class B : A { }
}

unittest
{
    // @rtInfo test
    assert(registeredClasses.length == 2);
    assert(registeredClasses[0] == "A" || registeredClasses[1] == "A");
    assert(registeredClasses[0] == "B" || registeredClasses[1] == "B");
}

jacob-carlborg · 2014-08-30T09:21:55Z

@yglukhov What's the status of this?

quickfur · 2014-09-15T21:49:00Z

ping @yglukhov

yglukhov · 2014-09-16T10:20:45Z

Ok, I've added a test to test_runnner.d. This is definitely not the best place for it, but I could not get it to link in object_.d.

MartinNowak · 2014-10-10T23:16:08Z

src/object.di

@@ -600,8 +600,53 @@ bool _xopCmp(in void* ptr, in void* ptr);
 void __ctfeWrite(T...)(auto ref T) {}
 void __ctfeWriteln(T...)(auto ref T values) { __ctfeWrite(values, "\n"); }

+private alias _Tuple(Args...) = Args;


Use core.internal.traits.TypeTuple instead.

… test/rtinfo.

yglukhov · 2014-10-14T09:35:51Z

Updated, rebased.

MartinNowak · 2014-10-18T16:45:46Z

Will close this for now as @rtInfo!MyCustomInfo is fairly pointless, because wherever you add @rtInfo!MyCustomInfo to a Type you might as well instantiate MyCustomInfo!Type yourself.

yglukhov · 2014-10-18T16:50:09Z

The point of the PR is to inspect subclasses in compile-time. Please, see the first comment in this PR.

MartinNowak · 2014-10-18T18:38:45Z

The point of the PR is to inspect subclasses in compile-time.

It's a huge machinery for this purpose and will probably slow down any compilation.
Have you considered other solutions (a mixin, CRTP)?
Also your proposal hasn't much to do with runtime information, you want to apply a template to a class hierarchy.

MartinNowak · 2014-10-18T18:45:13Z

Even better, you can use a base class constructor with a TemplateThisParameter to achieve what you want. http://dpaste.dzfl.pl/b66b4c18d582

ColdenCullen · 2014-10-23T21:55:04Z

@MartinNowak the downside to that is that it only works for one level of inheritance. So this only prints foo.

MartinNowak reviewed Apr 23, 2014
View reviewed changes

yglukhov mentioned this pull request Apr 24, 2014

Fix issue 10023 - Add rtInfo (or equivalent) to ModuleInfo. dlang/dmd#2271

Closed

MartinNowak reviewed Apr 24, 2014
View reviewed changes

src/object.di

template RTInfo(T)

{

template _instanciateRTInfoFromAttributes(Attrs...)

Copy link

Member

MartinNowak Apr 24, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's "instantiate" :).

MartinNowak reviewed Apr 24, 2014
View reviewed changes

MartinNowak reviewed Oct 10, 2014
View reviewed changes

uran and others added 6 commits October 14, 2014 12:04

Capability to get children compile-time info.

dc9bd18

Capability to get children compile-time info.

f12c1b7

Reimplemented custom RTInfo to use attributes.

ae0fec6

Fixed according to reviewer's notes.

d6d758b

Added test for cusrom RTInfo

3448323

Using core.internal.traits.TypeTuple instead of _Tuple. Test moved to…

55dc1e4

… test/rtinfo.

MartinNowak closed this Oct 18, 2014

Capability to get children compile-time info. #775

Capability to get children compile-time info. #775

Conversation

yglukhov commented Apr 23, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MartinNowak commented Apr 23, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yglukhov commented Apr 23, 2014

MartinNowak commented Apr 23, 2014

jacob-carlborg commented Apr 24, 2014

MartinNowak commented Apr 24, 2014

yglukhov commented Apr 24, 2014

jacob-carlborg commented Apr 24, 2014

MartinNowak commented Apr 24, 2014

Choose a reason for hiding this comment

yglukhov commented Apr 24, 2014

Choose a reason for hiding this comment

MartinNowak commented Apr 24, 2014

yglukhov commented Apr 25, 2014

MartinNowak commented Apr 25, 2014

jacob-carlborg commented Apr 26, 2014

yglukhov commented Apr 26, 2014

jacob-carlborg commented Apr 27, 2014

yglukhov commented Apr 27, 2014

jacob-carlborg commented Apr 27, 2014

rainers commented Apr 28, 2014

yglukhov commented Apr 28, 2014

rainers commented Apr 28, 2014

yglukhov commented Apr 28, 2014

yglukhov commented Apr 28, 2014

jacob-carlborg commented Apr 28, 2014

rainers commented Apr 28, 2014

yglukhov commented Apr 30, 2014

MartinNowak commented May 16, 2014

yglukhov commented Jun 4, 2014

jacob-carlborg commented Aug 30, 2014

quickfur commented Sep 15, 2014

yglukhov commented Sep 16, 2014

Choose a reason for hiding this comment

yglukhov commented Oct 14, 2014

MartinNowak commented Oct 18, 2014

yglukhov commented Oct 18, 2014

MartinNowak commented Oct 18, 2014

MartinNowak commented Oct 18, 2014

ColdenCullen commented Oct 23, 2014