Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Mixin gen #426

Closed
wants to merge 2 commits into from

9 participants

Martin Nowak Trass3r Hara Kenji Михаил Страшун Philippe Sigaud David Nadlinger Robert Clipsham Andrei Alexandrescu Andrej Mitrović
Martin Nowak
Collaborator

This pull adds a new command line option -M (-Mf, -Md).
When set string mixins are written to a separate file, which by default
carry a .mixin extension.
The logic for the files paths matches that of header generation.

The main benefits of adding this option are the simpler visibility of the
code vs. using pragma(msg) and fixing debug line information
which made it previously almost impossible to debug mixins.

I had to fix File::appendv and add File::append for POSIX.

Trass3r

Hmm what about template mixins?

Martin Nowak
Collaborator

Mixin templates are merely general templates with different instantiation scope
and have their code nicely defined in the sources.
Therefor it is no issue to debug/see the actual code.

Hara Kenji
Collaborator

I think we can use pragma(msg) for code printing.
If you want source position for printing, you can write a small utility like follows:

template CodePrinter(string code)
{
    @property string CodePrinter(string fn = __FILE__, size_t ln = __LINE__)()
    {
        pragma(msg, "mixin at ", fn, "(", cast(int)ln, ")");
        pragma(msg, "----\n", code, "\n----");
        return code;
    }
}
string gen()
{
    return "1 + 2";
}
struct S
{
    enum n = mixin(CodePrinter!(gen())());
}

output (in compilet time):

mixin at test.d(16)
----
1 + 2
----

What is the advantage of this proposal than a library way?

Martin Nowak
Collaborator

The advantage is that the debug output gets proper file and line information.
So when debugging a program you don't hit the macro wall.
Because it will not really be possible for any IDE tool to expand string mixins I don't
see any solution to this but to generate a file. Some recent C/C++ tools do expand macros
which is a huge help/feature in certain situations.

Trass3r

Does this also fix error message line numbers?

Martin Nowak
Collaborator

It could if the files were written before parsing and not afterwards.
Think I should change that.

Михаил Страшун

vote up, useful feature to switch on/off often, much more suited for compiler than library

Philippe Sigaud

So, no love for this, then?

David Nadlinger
Collaborator

Imho integrating this with the compiler makes a lot of sense, as it allows us to generate correct debug info for metaprogramming/string mixin-heavy code (this is a huge problem in practice). Only nitpick: The command line option description should state more clearly that this is just a debugging aid – as the average DMD user, I would wonder what a »mixin file« is…

Martin Nowak
Collaborator

The command line option description should state more clearly that this is just a debugging aid

Do you have any idea?

Does this also fix error message line numbers?

It does by now. The mixin code is appended before parsing and the AST has line number information pointing to the mixin file.

Robert Clipsham

How about "generate a file with all string mixins expanded (primarily useful for debugging)"?

Andrei Alexandrescu
Owner

(background: I'm doing a review of pull requests < 1000 as they tend to be controversial)

I think this has great promise, but we should rethink it. Large string mixins are arbitrarily difficult to compute (think parser generators, regexen, and such). What we really want is not to dump mixins in a file for future human examination, but develop a simple and coherent system for saving generated mixins across compilations - a sort of "precompiled mixins" if you wish. That way the compiler seeing ctRegex!"[0-9A-Z-a-z][0-9A-Z-a-z_-]*" will dump the generated string once and then reuse it unless it detect it must re-generate it.

That is a nontrivial task but, I think, one well worth thinking about if we want to scale up generative programming in D.

Brad Roberts braddr referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Martin Nowak
Collaborator

What we really want is not to dump mixins in a file for future human examination.

This pull is only about human examination. I made this because it fixes error messages and debug information. Both of which were already unvaluable tools for developing a lexer generator (https://github.com/dawgfoto/lexer).
I've experimented with way more complex DFA/LALR lexer/parser generators some month ago and doing that at compile time wouldn't have worked out without this.

unless it detect it must re-generate it

That sounds like Make or rdmd, doesn't it?

a sort of "precompiled mixins" if you wish

We could look into saving compressed, serialized parse trees at some point. This would benefit normal compilation too.
Otherwise I think we should rather focus on speeding up the compiler even though caching is never a bad idea.

9rnsr and others added some commits
Hara Kenji 9rnsr Merge pull request #1394 from dsagal/bug9068
Issue 9068 - fix compiler error when breaking from some labelled loops.
adcdfd7
Martin Nowak MartinNowak add -M command line option to output string mixins to extra file
 - this greatly simplifies to debug mixins
 - the usual -Mf -Md variants are implemented as well
 - needed to fix File::appendv and add File::append for
   POSIX
9086de5
Martin Nowak
Collaborator

The problem remains that debug info can't reference generated source code.

Andrei Alexandrescu
Owner

I'll close this for now pending further insights.

Andrei Alexandrescu andralex closed this
Andrej Mitrović
Collaborator

I think we should consider reopening this. As dawgfoto said, this pull is only about human examination (debugging), not about compilation caching.

AlexeyProkhin AlexeyProkhin referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
AlexeyProkhin AlexeyProkhin referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
AlexeyProkhin AlexeyProkhin referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
AlexeyProkhin AlexeyProkhin referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Dec 20, 2012
  1. Hara Kenji

    Merge pull request #1394 from dsagal/bug9068

    9rnsr authored
    Issue 9068 - fix compiler error when breaking from some labelled loops.
Commits on Dec 22, 2012
  1. Martin Nowak

    add -M command line option to output string mixins to extra file

    MartinNowak authored
     - this greatly simplifies to debug mixins
     - the usual -Mf -Md variants are implemented as well
     - needed to fix File::appendv and add File::append for
       POSIX
This page is out of date. Refresh to see the latest.
8 src/attrib.c
View
@@ -1544,9 +1544,15 @@ void CompileDeclaration::compileIt(Scope *sc)
{
se = se->toUTF8(sc);
Parser p(sc->module, (unsigned char *)se->string, se->len, 0);
- p.loc = loc;
+ if (global.params.doMxnGeneration)
+ p.loc = sc->module->importedFrom->writeMixin((unsigned char *)se->string, se->len, loc);
+ else
+ p.loc = loc;
+ const size_t linstart = p.loc.linnum;
p.nextToken();
decl = p.parseDeclDefs(0);
+ if (global.params.doMxnGeneration)
+ sc->module->importedFrom->mxnloc.linnum += p.loc.linnum - linstart;
if (p.token.value != TOKeof)
exp->error("incomplete mixin declaration (%s)", se->toChars());
}
8 src/expression.c
View
@@ -6568,10 +6568,16 @@ Expression *CompileExp::semantic(Scope *sc)
}
se = se->toUTF8(sc);
Parser p(sc->module, (unsigned char *)se->string, se->len, 0);
- p.loc = loc;
+ if (global.params.doMxnGeneration)
+ p.loc = sc->module->importedFrom->writeMixin((unsigned char *)se->string, se->len, loc);
+ else
+ p.loc = loc;
+ const size_t linstart = p.loc.linnum;
p.nextToken();
//printf("p.loc.linnum = %d\n", p.loc.linnum);
Expression *e = p.parseExpression();
+ if (global.params.doMxnGeneration)
+ sc->module->importedFrom->mxnloc.linnum += p.loc.linnum - linstart;
if (p.token.value != TOKeof)
{ error("incomplete mixin expression (%s)", se->toChars());
return new ErrorExp();
29 src/mars.c
View
@@ -62,6 +62,7 @@ Global::Global()
ddoc_ext = "ddoc";
json_ext = "json";
map_ext = "map";
+ mxn_ext = "mixin";
#if TARGET_WINDOS
obj_ext = "obj";
@@ -347,7 +348,10 @@ Usage:\n\
-inline do function inlining\n\
-Jpath where to look for string imports\n\
-Llinkerflag pass linkerflag to link\n\
- -lib generate library rather than object files\n"
+ -lib generate library rather than object files\n\
+ -M generate 'mixin' file\n\
+ -Mddirectory write 'mixin' file to directory\n\
+ -Mffilename write 'mixin' file to filename\n"
#if TARGET_LINUX || TARGET_OSX || TARGET_FREEBSD || TARGET_OPENBSD || TARGET_SOLARIS
" -m32 generate 32 bit code\n\
-m64 generate 64 bit code\n"
@@ -680,6 +684,29 @@ int tryMain(size_t argc, char *argv[])
goto Lerror;
}
}
+ else if (p[1] == 'M')
+ { global.params.doMxnGeneration = 1;
+ switch (p[2])
+ {
+ case 'd':
+ if (!p[3])
+ goto Lnoarg;
+ global.params.mxndir = p + 3;
+ break;
+
+ case 'f':
+ if (!p[3])
+ goto Lnoarg;
+ global.params.mxnname = p + 3;
+ break;
+
+ case 0:
+ break;
+
+ default:
+ goto Lerror;
+ }
+ }
else if (p[1] == 'X')
{ global.params.doXGeneration = 1;
switch (p[2])
11 src/mars.h
View
@@ -197,9 +197,13 @@ struct Param
char *docname; // write documentation file to docname
Strings *ddocfiles; // macro include files for Ddoc
- char doHdrGeneration; // process embedded documentation comments
- char *hdrdir; // write 'header' file to docdir directory
- char *hdrname; // write 'header' file to docname
+ char doHdrGeneration; // generate header files
+ char *hdrdir; // write 'header' file to hdrdir directory
+ char *hdrname; // write 'header' file to hdrname
+
+ char doMxnGeneration; // write mixins to file
+ char *mxndir; // write 'mixin' file to mxndir directory
+ char *mxnname; // write 'mixin' file to mxnname
char doXGeneration; // write JSON file
char *xfilename; // write JSON file to xfilename
@@ -258,6 +262,7 @@ struct Global
const char *hdr_ext; // for D 'header' import files
const char *json_ext; // for JSON files
const char *map_ext; // for .map files
+ const char *mxn_ext; // for D 'mixin' files
const char *copyright;
const char *written;
Strings *path; // Array of char*'s which form the import lookup path
58 src/module.c
View
@@ -907,6 +907,64 @@ void Module::gensymfile()
symfile->writev();
}
+/****************************************************
+ */
+
+Loc Module::writeMixin(unsigned char *str, size_t len, Loc loc)
+{
+ assert(global.params.doMxnGeneration);
+
+ if (!mxnfile)
+ {
+ FileName *mxnfilename;
+ char *argmxn;
+
+ if (global.params.mxnname)
+ argmxn = global.params.mxnname;
+ else if (global.params.preservePaths)
+ argmxn = (char *)arg;
+ else
+ argmxn = FileName::name((char *)arg);
+ if (!FileName::absolute(argmxn))
+ { //FileName::ensurePathExists(global.params.hdrdir);
+ argmxn = FileName::combine(global.params.mxndir, argmxn);
+ }
+ if (global.params.mxnname)
+ mxnfilename = new FileName(argmxn);
+ else
+ mxnfilename = FileName::forceExt(argmxn, global.mxn_ext);
+
+ if (mxnfilename->equals(srcfile->name))
+ { error("Source file and 'mixin' file have same name '%s'", srcfile->name->str);
+ fatal();
+ }
+
+ char *pt = FileName::path(mxnfilename->str);
+ if (*pt)
+ FileName::ensurePathExists(pt);
+ mem.free(pt);
+
+ mxnfile = new File(mxnfilename);
+ mxnfile->remove();
+ mxnloc = Loc(1);
+ mxnloc.filename = mxnfilename->str;
+ }
+ OutBuffer buf;
+
+ buf.printf("// mixin at %s", loc.toChars());
+ buf.writenl();
+ ++mxnloc.linnum;
+ Loc start = mxnloc;
+ buf.write(str, len);
+ buf.writenl();
+ ++mxnloc.linnum;
+
+ mxnfile->setbuffer(buf.data, buf.offset);
+ mxnfile->appendv();
+
+ return start;
+}
+
/**********************************
* Determine if we need to generate an instance of ModuleInfo
* for this Module.
3  src/module.h
View
@@ -64,6 +64,8 @@ struct Module : Package
File *hdrfile; // 'header' file
File *symfile; // output symbol file
File *docfile; // output documentation file
+ File *mxnfile; // 'mixin' file
+ Loc mxnloc; // location in mixin file
unsigned errors; // if any errors in file
unsigned numlines; // number of lines in source file
int isDocFile; // if it is a documentation input file, not D source
@@ -131,6 +133,7 @@ struct Module : Package
void genobjfile(int multiobj);
void gensymfile();
void gendocfile();
+ Loc writeMixin(unsigned char* str, size_t len, Loc loc);
int needModuleInfo();
Dsymbol *search(Loc loc, Identifier *ident, int flags);
Dsymbol *symtabInsert(Dsymbol *s);
84 src/root/root.c
View
@@ -1165,37 +1165,7 @@ int File::mmread()
int File::write()
{
#if POSIX
- int fd;
- ssize_t numwritten;
- char *name;
-
- name = this->name->toChars();
- fd = open(name, O_CREAT | O_WRONLY | O_TRUNC, 0644);
- if (fd == -1)
- goto err;
-
- numwritten = ::write(fd, buffer, len);
- if (len != numwritten)
- goto err2;
-
- if (close(fd) == -1)
- goto err;
-
- if (touchtime)
- { struct utimbuf ubuf;
-
- ubuf.actime = ((struct stat *)touchtime)->st_atime;
- ubuf.modtime = ((struct stat *)touchtime)->st_mtime;
- if (utime(name, &ubuf))
- goto err;
- }
- return 0;
-
-err2:
- close(fd);
- ::remove(name);
-err:
- return 1;
+ return writePosix(false);
#elif _WIN32
HANDLE h;
DWORD numwritten;
@@ -1239,7 +1209,7 @@ int File::write()
int File::append()
{
#if POSIX
- return 1;
+ return writePosix(true);
#elif _WIN32
HANDLE h;
DWORD numwritten;
@@ -1280,6 +1250,54 @@ int File::append()
#endif
}
+/*********************************************
+ * Writes or Append to a file.
+ * Returns:
+ * 0 success
+ */
+
+#if POSIX
+int File::writePosix(bool append)
+{
+ int fd;
+ ssize_t numwritten;
+ char *name;
+ int flags = O_CREAT | O_WRONLY;
+
+ if (append)
+ flags |= O_APPEND;
+ else
+ flags |= O_TRUNC;
+ name = this->name->toChars();
+ fd = open(name, flags, 0644);
+ if (fd == -1)
+ goto err;
+
+ numwritten = ::write(fd, buffer, len);
+ if (len != numwritten)
+ goto err2;
+
+ if (close(fd) == -1)
+ goto err;
+
+ if (touchtime)
+ { struct utimbuf ubuf;
+
+ ubuf.actime = ((struct stat *)touchtime)->st_atime;
+ ubuf.modtime = ((struct stat *)touchtime)->st_mtime;
+ if (utime(name, &ubuf))
+ goto err;
+ }
+ return 0;
+
+err2:
+ close(fd);
+ ::remove(name);
+err:
+ return 1;
+}
+#endif
+
/**************************************
*/
@@ -1306,7 +1324,7 @@ void File::writev()
void File::appendv()
{
- if (write())
+ if (append())
error("Error appending to file '%s'",name->toChars());
}
7 src/root/root.h
View
@@ -162,6 +162,7 @@ struct File : Object
void mmreadv();
+
/* Write file, return !=0 if error
*/
@@ -196,6 +197,12 @@ struct File : Object
* 2: directory
*/
+#if POSIX
+ /* POSIX implementation of write and append
+ */
+ int writePosix(bool append);
+#endif
+
int exists();
/* Given wildcard filespec, return an array of
9 src/statement.c
View
@@ -29,6 +29,7 @@
#include "template.h"
#include "attrib.h"
#include "import.h"
+#include "module.h"
extern int os_critsecsize32();
extern int os_critsecsize64();
@@ -439,7 +440,11 @@ Statements *CompileStatement::flatten(Scope *sc)
}
se = se->toUTF8(sc);
Parser p(sc->module, (unsigned char *)se->string, se->len, 0);
- p.loc = loc;
+ if (global.params.doMxnGeneration)
+ p.loc = sc->module->importedFrom->writeMixin((unsigned char *)se->string, se->len, loc);
+ else
+ p.loc = loc;
+ const size_t linstart = p.loc.linnum;
p.nextToken();
Statements *a = new Statements();
@@ -449,6 +454,8 @@ Statements *CompileStatement::flatten(Scope *sc)
if (s) // if no parsing errors
a->push(s);
}
+ if (global.params.doMxnGeneration)
+ sc->module->importedFrom->mxnloc.linnum += p.loc.linnum - linstart;
return a;
}
9 test/compilable/extra-files/mixin-postscript.sh
View
@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+
+diff --strip-trailing-cr compilable/extra-files/mixin.mixin ${RESULTS_DIR}/compilable/mixin.mixin
+if [ $? -ne 0 ]; then
+ exit 1;
+fi
+
+rm ${RESULTS_DIR}/compilable/mixin.mixin
+
37 test/compilable/extra-files/mixin.mixin
View
@@ -0,0 +1,37 @@
+// mixin at compilable/mixin.d(64)
+
+int bar2()
+{
+ mixin Foo2;
+ return result();
+}
+
+// mixin at compilable/mixin.d(9)
+5 + 3
+// mixin at compilable/mixin.d(10)
+int a1 = 5;
+int b1 = 10;
+auto res1 = a1 * b1;
+
+// mixin at compilable/mixin.d(16)
+
+ int a2 = 5;
+ int b2 = 10;
+ auto res2 = a2 * b2;
+
+// mixin at compilable/mixin.d(41)
+5 + 3
+// mixin at compilable/mixin.d(42)
+int s1a = 5;
+int s1b = 10;
+auto s1res = s1a * s1b;
+
+// mixin at compilable/mixin.d(61)
+
+ int result()
+ {
+ auto tmp = a + mixin(bar2nested2());
+ return tmp;
+ };
+// mixin at test_results/compilable/mixin.mixin(33)
+getb
96 test/compilable/mixin.d
View
@@ -0,0 +1,96 @@
+// PERMUTE_ARGS:
+// REQUIRED_ARGS: -M -Mdtest_results/compilable
+// POST_SCRIPT: compilable/extra-files/mixin-postscript.sh
+
+module foo.bar;
+
+void foo()
+{
+ auto a = mixin("5 + 3");
+ mixin(
+ "int a1 = 5;\n"
+ "int b1 = 10;\n"
+ "auto res1 = a1 * b1;\n"
+ );
+
+ mixin("
+ int a2 = 5;
+ int b2 = 10;
+ auto res2 = a2 * b2;
+");
+}
+
+string exp1()
+{
+ return "5 + 3";
+}
+
+string stmt1()
+{
+ string s = "int s1a = 5;\n";
+ s ~= "int s1b = 10;\n";
+ s ~= "auto s1res = s1a * s1b;\n";
+ return s;
+}
+
+enum e1 = exp1();
+enum s1 = stmt1();
+
+void bar()
+{
+ auto e1res = mixin(e1);
+ mixin(s1);
+}
+
+// Only generate output for string mixins
+mixin template Foo()
+{
+ int x = 5;
+}
+
+mixin Foo;
+
+struct Bar {
+ mixin Foo;
+}
+
+mixin template Foo2()
+{
+ int a = 5;
+ int b = 3;
+ mixin(bar2nested());
+}
+
+mixin("
+int bar2()
+{
+ mixin Foo2;
+ return result();
+}
+");
+
+string bar2nested()
+{
+ return "
+ int result()
+ {
+ auto tmp = a + mixin(bar2nested2());
+ return tmp;
+ };";
+}
+
+string bar2nested2()
+{
+ return "getb";
+}
+
+@property int getb()
+{
+ return 6;
+}
+
+void main()
+{
+ bar();
+ bar2();
+}
Something went wrong with that request. Please try again.