Mixin gen #426

Closed
wants to merge 2 commits into
from

Conversation

Projects
None yet
8 participants
@MartinNowak
Member

MartinNowak commented Oct 3, 2011

This pull adds a new command line option -M (-Mf, -Md).
When set string mixins are written to a separate file, which by default
carry a .mixin extension.
The logic for the files paths matches that of header generation.

The main benefits of adding this option are the simpler visibility of the
code vs. using pragma(msg) and fixing debug line information
which made it previously almost impossible to debug mixins.

I had to fix File::appendv and add File::append for POSIX.

@Trass3r

This comment has been minimized.

Show comment
Hide comment
@Trass3r

Trass3r Oct 4, 2011

Contributor

Hmm what about template mixins?

Contributor

Trass3r commented Oct 4, 2011

Hmm what about template mixins?

@MartinNowak

This comment has been minimized.

Show comment
Hide comment
@MartinNowak

MartinNowak Oct 5, 2011

Member

Mixin templates are merely general templates with different instantiation scope
and have their code nicely defined in the sources.
Therefor it is no issue to debug/see the actual code.

Member

MartinNowak commented Oct 5, 2011

Mixin templates are merely general templates with different instantiation scope
and have their code nicely defined in the sources.
Therefor it is no issue to debug/see the actual code.

@9rnsr

This comment has been minimized.

Show comment
Hide comment
@9rnsr

9rnsr Oct 8, 2011

Member

I think we can use pragma(msg) for code printing.
If you want source position for printing, you can write a small utility like follows:

template CodePrinter(string code)
{
    @property string CodePrinter(string fn = __FILE__, size_t ln = __LINE__)()
    {
        pragma(msg, "mixin at ", fn, "(", cast(int)ln, ")");
        pragma(msg, "----\n", code, "\n----");
        return code;
    }
}
string gen()
{
    return "1 + 2";
}
struct S
{
    enum n = mixin(CodePrinter!(gen())());
}

output (in compilet time):

mixin at test.d(16)
----
1 + 2
----

What is the advantage of this proposal than a library way?

Member

9rnsr commented Oct 8, 2011

I think we can use pragma(msg) for code printing.
If you want source position for printing, you can write a small utility like follows:

template CodePrinter(string code)
{
    @property string CodePrinter(string fn = __FILE__, size_t ln = __LINE__)()
    {
        pragma(msg, "mixin at ", fn, "(", cast(int)ln, ")");
        pragma(msg, "----\n", code, "\n----");
        return code;
    }
}
string gen()
{
    return "1 + 2";
}
struct S
{
    enum n = mixin(CodePrinter!(gen())());
}

output (in compilet time):

mixin at test.d(16)
----
1 + 2
----

What is the advantage of this proposal than a library way?

@MartinNowak

This comment has been minimized.

Show comment
Hide comment
@MartinNowak

MartinNowak Oct 9, 2011

Member

The advantage is that the debug output gets proper file and line information.
So when debugging a program you don't hit the macro wall.
Because it will not really be possible for any IDE tool to expand string mixins I don't
see any solution to this but to generate a file. Some recent C/C++ tools do expand macros
which is a huge help/feature in certain situations.

Member

MartinNowak commented Oct 9, 2011

The advantage is that the debug output gets proper file and line information.
So when debugging a program you don't hit the macro wall.
Because it will not really be possible for any IDE tool to expand string mixins I don't
see any solution to this but to generate a file. Some recent C/C++ tools do expand macros
which is a huge help/feature in certain situations.

@Trass3r

This comment has been minimized.

Show comment
Hide comment
@Trass3r

Trass3r Jan 6, 2012

Contributor

Does this also fix error message line numbers?

Contributor

Trass3r commented Jan 6, 2012

Does this also fix error message line numbers?

@MartinNowak

This comment has been minimized.

Show comment
Hide comment
@MartinNowak

MartinNowak Jan 6, 2012

Member

It could if the files were written before parsing and not afterwards.
Think I should change that.

Member

MartinNowak commented Jan 6, 2012

It could if the files were written before parsing and not afterwards.
Think I should change that.

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Jan 9, 2012

vote up, useful feature to switch on/off often, much more suited for compiler than library

vote up, useful feature to switch on/off often, much more suited for compiler than library

@PhilippeSigaud

This comment has been minimized.

Show comment
Hide comment
@PhilippeSigaud

PhilippeSigaud Jul 23, 2012

So, no love for this, then?

So, no love for this, then?

@klickverbot

This comment has been minimized.

Show comment
Hide comment
@klickverbot

klickverbot Jul 23, 2012

Member

Imho integrating this with the compiler makes a lot of sense, as it allows us to generate correct debug info for metaprogramming/string mixin-heavy code (this is a huge problem in practice). Only nitpick: The command line option description should state more clearly that this is just a debugging aid – as the average DMD user, I would wonder what a »mixin file« is…

Member

klickverbot commented Jul 23, 2012

Imho integrating this with the compiler makes a lot of sense, as it allows us to generate correct debug info for metaprogramming/string mixin-heavy code (this is a huge problem in practice). Only nitpick: The command line option description should state more clearly that this is just a debugging aid – as the average DMD user, I would wonder what a »mixin file« is…

@MartinNowak

This comment has been minimized.

Show comment
Hide comment
@MartinNowak

MartinNowak Jul 23, 2012

Member

The command line option description should state more clearly that this is just a debugging aid

Do you have any idea?

Does this also fix error message line numbers?

It does by now. The mixin code is appended before parsing and the AST has line number information pointing to the mixin file.

Member

MartinNowak commented Jul 23, 2012

The command line option description should state more clearly that this is just a debugging aid

Do you have any idea?

Does this also fix error message line numbers?

It does by now. The mixin code is appended before parsing and the AST has line number information pointing to the mixin file.

@mrmonday

This comment has been minimized.

Show comment
Hide comment
@mrmonday

mrmonday Jul 23, 2012

Contributor

How about "generate a file with all string mixins expanded (primarily useful for debugging)"?

Contributor

mrmonday commented Jul 23, 2012

How about "generate a file with all string mixins expanded (primarily useful for debugging)"?

@andralex

This comment has been minimized.

Show comment
Hide comment
@andralex

andralex Sep 24, 2012

Member

(background: I'm doing a review of pull requests < 1000 as they tend to be controversial)

I think this has great promise, but we should rethink it. Large string mixins are arbitrarily difficult to compute (think parser generators, regexen, and such). What we really want is not to dump mixins in a file for future human examination, but develop a simple and coherent system for saving generated mixins across compilations - a sort of "precompiled mixins" if you wish. That way the compiler seeing ctRegex!"[0-9A-Z-a-z][0-9A-Z-a-z_-]*" will dump the generated string once and then reuse it unless it detect it must re-generate it.

That is a nontrivial task but, I think, one well worth thinking about if we want to scale up generative programming in D.

Member

andralex commented Sep 24, 2012

(background: I'm doing a review of pull requests < 1000 as they tend to be controversial)

I think this has great promise, but we should rethink it. Large string mixins are arbitrarily difficult to compute (think parser generators, regexen, and such). What we really want is not to dump mixins in a file for future human examination, but develop a simple and coherent system for saving generated mixins across compilations - a sort of "precompiled mixins" if you wish. That way the compiler seeing ctRegex!"[0-9A-Z-a-z][0-9A-Z-a-z_-]*" will dump the generated string once and then reuse it unless it detect it must re-generate it.

That is a nontrivial task but, I think, one well worth thinking about if we want to scale up generative programming in D.

@MartinNowak

This comment has been minimized.

Show comment
Hide comment
@MartinNowak

MartinNowak Dec 16, 2012

Member

What we really want is not to dump mixins in a file for future human examination.

This pull is only about human examination. I made this because it fixes error messages and debug information. Both of which were already unvaluable tools for developing a lexer generator (https://github.com/dawgfoto/lexer).
I've experimented with way more complex DFA/LALR lexer/parser generators some month ago and doing that at compile time wouldn't have worked out without this.

unless it detect it must re-generate it

That sounds like Make or rdmd, doesn't it?

a sort of "precompiled mixins" if you wish

We could look into saving compressed, serialized parse trees at some point. This would benefit normal compilation too.
Otherwise I think we should rather focus on speeding up the compiler even though caching is never a bad idea.

Member

MartinNowak commented Dec 16, 2012

What we really want is not to dump mixins in a file for future human examination.

This pull is only about human examination. I made this because it fixes error messages and debug information. Both of which were already unvaluable tools for developing a lexer generator (https://github.com/dawgfoto/lexer).
I've experimented with way more complex DFA/LALR lexer/parser generators some month ago and doing that at compile time wouldn't have worked out without this.

unless it detect it must re-generate it

That sounds like Make or rdmd, doesn't it?

a sort of "precompiled mixins" if you wish

We could look into saving compressed, serialized parse trees at some point. This would benefit normal compilation too.
Otherwise I think we should rather focus on speeding up the compiler even though caching is never a bad idea.

9rnsr and others added some commits Dec 20, 2012

Merge pull request #1394 from dsagal/bug9068
Issue 9068 - fix compiler error when breaking from some labelled loops.
add -M command line option to output string mixins to extra file
 - this greatly simplifies to debug mixins
 - the usual -Mf -Md variants are implemented as well
 - needed to fix File::appendv and add File::append for
   POSIX
@MartinNowak

This comment has been minimized.

Show comment
Hide comment
@MartinNowak

MartinNowak Apr 7, 2013

Member

The problem remains that debug info can't reference generated source code.

Member

MartinNowak commented Apr 7, 2013

The problem remains that debug info can't reference generated source code.

@andralex

This comment has been minimized.

Show comment
Hide comment
@andralex

andralex May 12, 2013

Member

I'll close this for now pending further insights.

Member

andralex commented May 12, 2013

I'll close this for now pending further insights.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Nov 19, 2013

I think we should consider reopening this. As dawgfoto said, this pull is only about human examination (debugging), not about compilation caching.

ghost commented Nov 19, 2013

I think we should consider reopening this. As dawgfoto said, this pull is only about human examination (debugging), not about compilation caching.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment