Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Mixin gen #426

Closed
wants to merge 2 commits into from

9 participants

Martin Nowak Trass3r Hara Kenji Михаил Страшун Philippe Sigaud David Nadlinger Robert Clipsham Andrei Alexandrescu Andrej Mitrovic
Martin Nowak
Collaborator

This pull adds a new command line option -M (-Mf, -Md).
When set string mixins are written to a separate file, which by default
carry a .mixin extension.
The logic for the files paths matches that of header generation.

The main benefits of adding this option are the simpler visibility of the
code vs. using pragma(msg) and fixing debug line information
which made it previously almost impossible to debug mixins.

I had to fix File::appendv and add File::append for POSIX.

Trass3r

Hmm what about template mixins?

Martin Nowak
Collaborator

Mixin templates are merely general templates with different instantiation scope
and have their code nicely defined in the sources.
Therefor it is no issue to debug/see the actual code.

Hara Kenji
Collaborator

I think we can use pragma(msg) for code printing.
If you want source position for printing, you can write a small utility like follows:

template CodePrinter(string code)
{
    @property string CodePrinter(string fn = __FILE__, size_t ln = __LINE__)()
    {
        pragma(msg, "mixin at ", fn, "(", cast(int)ln, ")");
        pragma(msg, "----\n", code, "\n----");
        return code;
    }
}
string gen()
{
    return "1 + 2";
}
struct S
{
    enum n = mixin(CodePrinter!(gen())());
}

output (in compilet time):

mixin at test.d(16)
----
1 + 2
----

What is the advantage of this proposal than a library way?

Martin Nowak
Collaborator

The advantage is that the debug output gets proper file and line information.
So when debugging a program you don't hit the macro wall.
Because it will not really be possible for any IDE tool to expand string mixins I don't
see any solution to this but to generate a file. Some recent C/C++ tools do expand macros
which is a huge help/feature in certain situations.

Trass3r

Does this also fix error message line numbers?

Martin Nowak
Collaborator

It could if the files were written before parsing and not afterwards.
Think I should change that.

Михаил Страшун

vote up, useful feature to switch on/off often, much more suited for compiler than library

Philippe Sigaud

So, no love for this, then?

David Nadlinger
Collaborator

Imho integrating this with the compiler makes a lot of sense, as it allows us to generate correct debug info for metaprogramming/string mixin-heavy code (this is a huge problem in practice). Only nitpick: The command line option description should state more clearly that this is just a debugging aid – as the average DMD user, I would wonder what a »mixin file« is…

Martin Nowak
Collaborator

The command line option description should state more clearly that this is just a debugging aid

Do you have any idea?

Does this also fix error message line numbers?

It does by now. The mixin code is appended before parsing and the AST has line number information pointing to the mixin file.

Robert Clipsham

How about "generate a file with all string mixins expanded (primarily useful for debugging)"?

Andrei Alexandrescu
Owner

(background: I'm doing a review of pull requests < 1000 as they tend to be controversial)

I think this has great promise, but we should rethink it. Large string mixins are arbitrarily difficult to compute (think parser generators, regexen, and such). What we really want is not to dump mixins in a file for future human examination, but develop a simple and coherent system for saving generated mixins across compilations - a sort of "precompiled mixins" if you wish. That way the compiler seeing ctRegex!"[0-9A-Z-a-z][0-9A-Z-a-z_-]*" will dump the generated string once and then reuse it unless it detect it must re-generate it.

That is a nontrivial task but, I think, one well worth thinking about if we want to scale up generative programming in D.

Brad Roberts braddr referenced this pull request from a commit October 21, 2012
Commit has since been removed from the repository and is no longer available.
Martin Nowak
Collaborator

What we really want is not to dump mixins in a file for future human examination.

This pull is only about human examination. I made this because it fixes error messages and debug information. Both of which were already unvaluable tools for developing a lexer generator (https://github.com/dawgfoto/lexer).
I've experimented with way more complex DFA/LALR lexer/parser generators some month ago and doing that at compile time wouldn't have worked out without this.

unless it detect it must re-generate it

That sounds like Make or rdmd, doesn't it?

a sort of "precompiled mixins" if you wish

We could look into saving compressed, serialized parse trees at some point. This would benefit normal compilation too.
Otherwise I think we should rather focus on speeding up the compiler even though caching is never a bad idea.

and others added some commits December 20, 2012
Hara Kenji Merge pull request #1394 from dsagal/bug9068
Issue 9068 - fix compiler error when breaking from some labelled loops.
adcdfd7
Martin Nowak add -M command line option to output string mixins to extra file
 - this greatly simplifies to debug mixins
 - the usual -Mf -Md variants are implemented as well
 - needed to fix File::appendv and add File::append for
   POSIX
9086de5
Martin Nowak
Collaborator

The problem remains that debug info can't reference generated source code.

Andrei Alexandrescu
Owner

I'll close this for now pending further insights.

Andrei Alexandrescu andralex closed this May 12, 2013
Andrej Mitrovic
Collaborator

I think we should consider reopening this. As dawgfoto said, this pull is only about human examination (debugging), not about compilation caching.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 2 unique commits by 2 authors.

Dec 20, 2012
Hara Kenji Merge pull request #1394 from dsagal/bug9068
Issue 9068 - fix compiler error when breaking from some labelled loops.
adcdfd7
Dec 22, 2012
Martin Nowak add -M command line option to output string mixins to extra file
 - this greatly simplifies to debug mixins
 - the usual -Mf -Md variants are implemented as well
 - needed to fix File::appendv and add File::append for
   POSIX
9086de5
This page is out of date. Refresh to see the latest.
8  src/attrib.c
@@ -1544,9 +1544,15 @@ void CompileDeclaration::compileIt(Scope *sc)
1544 1544
     {
1545 1545
         se = se->toUTF8(sc);
1546 1546
         Parser p(sc->module, (unsigned char *)se->string, se->len, 0);
1547  
-        p.loc = loc;
  1547
+        if (global.params.doMxnGeneration)
  1548
+            p.loc = sc->module->importedFrom->writeMixin((unsigned char *)se->string, se->len, loc);
  1549
+        else
  1550
+            p.loc = loc;
  1551
+        const size_t linstart = p.loc.linnum;
1548 1552
         p.nextToken();
1549 1553
         decl = p.parseDeclDefs(0);
  1554
+        if (global.params.doMxnGeneration)
  1555
+            sc->module->importedFrom->mxnloc.linnum += p.loc.linnum - linstart;
1550 1556
         if (p.token.value != TOKeof)
1551 1557
             exp->error("incomplete mixin declaration (%s)", se->toChars());
1552 1558
     }
8  src/expression.c
@@ -6568,10 +6568,16 @@ Expression *CompileExp::semantic(Scope *sc)
6568 6568
     }
6569 6569
     se = se->toUTF8(sc);
6570 6570
     Parser p(sc->module, (unsigned char *)se->string, se->len, 0);
6571  
-    p.loc = loc;
  6571
+    if (global.params.doMxnGeneration)
  6572
+        p.loc = sc->module->importedFrom->writeMixin((unsigned char *)se->string, se->len, loc);
  6573
+    else
  6574
+        p.loc = loc;
  6575
+    const size_t linstart = p.loc.linnum;
6572 6576
     p.nextToken();
6573 6577
     //printf("p.loc.linnum = %d\n", p.loc.linnum);
6574 6578
     Expression *e = p.parseExpression();
  6579
+    if (global.params.doMxnGeneration)
  6580
+        sc->module->importedFrom->mxnloc.linnum += p.loc.linnum - linstart;
6575 6581
     if (p.token.value != TOKeof)
6576 6582
     {   error("incomplete mixin expression (%s)", se->toChars());
6577 6583
         return new ErrorExp();
29  src/mars.c
@@ -62,6 +62,7 @@ Global::Global()
62 62
     ddoc_ext = "ddoc";
63 63
     json_ext = "json";
64 64
     map_ext  = "map";
  65
+    mxn_ext  = "mixin";
65 66
 
66 67
 #if TARGET_WINDOS
67 68
     obj_ext  = "obj";
@@ -347,7 +348,10 @@ Usage:\n\
347 348
   -inline        do function inlining\n\
348 349
   -Jpath         where to look for string imports\n\
349 350
   -Llinkerflag   pass linkerflag to link\n\
350  
-  -lib           generate library rather than object files\n"
  351
+  -lib           generate library rather than object files\n\
  352
+  -M             generate 'mixin' file\n\
  353
+  -Mddirectory   write 'mixin' file to directory\n\
  354
+  -Mffilename    write 'mixin' file to filename\n"
351 355
 #if TARGET_LINUX || TARGET_OSX || TARGET_FREEBSD || TARGET_OPENBSD || TARGET_SOLARIS
352 356
 "  -m32           generate 32 bit code\n\
353 357
   -m64           generate 64 bit code\n"
@@ -680,6 +684,29 @@ int tryMain(size_t argc, char *argv[])
680 684
                         goto Lerror;
681 685
                 }
682 686
             }
  687
+            else if (p[1] == 'M')
  688
+            {   global.params.doMxnGeneration = 1;
  689
+                switch (p[2])
  690
+                {
  691
+                    case 'd':
  692
+                        if (!p[3])
  693
+                            goto Lnoarg;
  694
+                        global.params.mxndir = p + 3;
  695
+                        break;
  696
+
  697
+                    case 'f':
  698
+                        if (!p[3])
  699
+                            goto Lnoarg;
  700
+                        global.params.mxnname = p + 3;
  701
+                        break;
  702
+
  703
+                    case 0:
  704
+                        break;
  705
+
  706
+                    default:
  707
+                        goto Lerror;
  708
+                }
  709
+            }
683 710
             else if (p[1] == 'X')
684 711
             {   global.params.doXGeneration = 1;
685 712
                 switch (p[2])
11  src/mars.h
@@ -197,9 +197,13 @@ struct Param
197 197
     char *docname;      // write documentation file to docname
198 198
     Strings *ddocfiles;   // macro include files for Ddoc
199 199
 
200  
-    char doHdrGeneration;       // process embedded documentation comments
201  
-    char *hdrdir;               // write 'header' file to docdir directory
202  
-    char *hdrname;              // write 'header' file to docname
  200
+    char doHdrGeneration;       // generate header files
  201
+    char *hdrdir;               // write 'header' file to hdrdir directory
  202
+    char *hdrname;              // write 'header' file to hdrname
  203
+
  204
+    char doMxnGeneration;       // write mixins to file
  205
+    char *mxndir;               // write 'mixin' file to mxndir directory
  206
+    char *mxnname;              // write 'mixin' file to mxnname
203 207
 
204 208
     char doXGeneration;         // write JSON file
205 209
     char *xfilename;            // write JSON file to xfilename
@@ -258,6 +262,7 @@ struct Global
258 262
     const char *hdr_ext;        // for D 'header' import files
259 263
     const char *json_ext;       // for JSON files
260 264
     const char *map_ext;        // for .map files
  265
+    const char *mxn_ext;        // for D 'mixin' files
261 266
     const char *copyright;
262 267
     const char *written;
263 268
     Strings *path;        // Array of char*'s which form the import lookup path
58  src/module.c
@@ -907,6 +907,64 @@ void Module::gensymfile()
907 907
     symfile->writev();
908 908
 }
909 909
 
  910
+/****************************************************
  911
+ */
  912
+
  913
+Loc Module::writeMixin(unsigned char *str, size_t len, Loc loc)
  914
+{
  915
+    assert(global.params.doMxnGeneration);
  916
+
  917
+    if (!mxnfile)
  918
+    {
  919
+        FileName *mxnfilename;
  920
+        char *argmxn;
  921
+
  922
+        if (global.params.mxnname)
  923
+            argmxn = global.params.mxnname;
  924
+        else if (global.params.preservePaths)
  925
+            argmxn = (char *)arg;
  926
+        else
  927
+            argmxn = FileName::name((char *)arg);
  928
+        if (!FileName::absolute(argmxn))
  929
+        {   //FileName::ensurePathExists(global.params.hdrdir);
  930
+            argmxn = FileName::combine(global.params.mxndir, argmxn);
  931
+        }
  932
+        if (global.params.mxnname)
  933
+            mxnfilename = new FileName(argmxn);
  934
+        else
  935
+            mxnfilename = FileName::forceExt(argmxn, global.mxn_ext);
  936
+
  937
+        if (mxnfilename->equals(srcfile->name))
  938
+        {   error("Source file and 'mixin' file have same name '%s'", srcfile->name->str);
  939
+            fatal();
  940
+        }
  941
+
  942
+        char *pt = FileName::path(mxnfilename->str);
  943
+        if (*pt)
  944
+            FileName::ensurePathExists(pt);
  945
+        mem.free(pt);
  946
+
  947
+        mxnfile = new File(mxnfilename);
  948
+        mxnfile->remove();
  949
+        mxnloc = Loc(1);
  950
+        mxnloc.filename = mxnfilename->str;
  951
+    }
  952
+    OutBuffer buf;
  953
+
  954
+    buf.printf("// mixin at %s", loc.toChars());
  955
+    buf.writenl();
  956
+    ++mxnloc.linnum;
  957
+    Loc start = mxnloc;
  958
+    buf.write(str, len);
  959
+    buf.writenl();
  960
+    ++mxnloc.linnum;
  961
+
  962
+    mxnfile->setbuffer(buf.data, buf.offset);
  963
+    mxnfile->appendv();
  964
+
  965
+    return start;
  966
+}
  967
+
910 968
 /**********************************
911 969
  * Determine if we need to generate an instance of ModuleInfo
912 970
  * for this Module.
3  src/module.h
@@ -64,6 +64,8 @@ struct Module : Package
64 64
     File *hdrfile;      // 'header' file
65 65
     File *symfile;      // output symbol file
66 66
     File *docfile;      // output documentation file
  67
+    File *mxnfile;      // 'mixin' file
  68
+    Loc mxnloc;         // location in mixin file
67 69
     unsigned errors;    // if any errors in file
68 70
     unsigned numlines;  // number of lines in source file
69 71
     int isDocFile;      // if it is a documentation input file, not D source
@@ -131,6 +133,7 @@ struct Module : Package
131 133
     void genobjfile(int multiobj);
132 134
     void gensymfile();
133 135
     void gendocfile();
  136
+    Loc writeMixin(unsigned char* str, size_t len, Loc loc);
134 137
     int needModuleInfo();
135 138
     Dsymbol *search(Loc loc, Identifier *ident, int flags);
136 139
     Dsymbol *symtabInsert(Dsymbol *s);
84  src/root/root.c
@@ -1165,37 +1165,7 @@ int File::mmread()
1165 1165
 int File::write()
1166 1166
 {
1167 1167
 #if POSIX
1168  
-    int fd;
1169  
-    ssize_t numwritten;
1170  
-    char *name;
1171  
-
1172  
-    name = this->name->toChars();
1173  
-    fd = open(name, O_CREAT | O_WRONLY | O_TRUNC, 0644);
1174  
-    if (fd == -1)
1175  
-        goto err;
1176  
-
1177  
-    numwritten = ::write(fd, buffer, len);
1178  
-    if (len != numwritten)
1179  
-        goto err2;
1180  
-
1181  
-    if (close(fd) == -1)
1182  
-        goto err;
1183  
-
1184  
-    if (touchtime)
1185  
-    {   struct utimbuf ubuf;
1186  
-
1187  
-        ubuf.actime = ((struct stat *)touchtime)->st_atime;
1188  
-        ubuf.modtime = ((struct stat *)touchtime)->st_mtime;
1189  
-        if (utime(name, &ubuf))
1190  
-            goto err;
1191  
-    }
1192  
-    return 0;
1193  
-
1194  
-err2:
1195  
-    close(fd);
1196  
-    ::remove(name);
1197  
-err:
1198  
-    return 1;
  1168
+    return writePosix(false);
1199 1169
 #elif _WIN32
1200 1170
     HANDLE h;
1201 1171
     DWORD numwritten;
@@ -1239,7 +1209,7 @@ int File::write()
1239 1209
 int File::append()
1240 1210
 {
1241 1211
 #if POSIX
1242  
-    return 1;
  1212
+    return writePosix(true);
1243 1213
 #elif _WIN32
1244 1214
     HANDLE h;
1245 1215
     DWORD numwritten;
@@ -1280,6 +1250,54 @@ int File::append()
1280 1250
 #endif
1281 1251
 }
1282 1252
 
  1253
+/*********************************************
  1254
+ * Writes or Append to a file.
  1255
+ * Returns:
  1256
+ *      0       success
  1257
+ */
  1258
+
  1259
+#if POSIX
  1260
+int File::writePosix(bool append)
  1261
+{
  1262
+    int fd;
  1263
+    ssize_t numwritten;
  1264
+    char *name;
  1265
+    int flags = O_CREAT | O_WRONLY;
  1266
+
  1267
+    if (append)
  1268
+        flags |= O_APPEND;
  1269
+    else
  1270
+        flags |= O_TRUNC;
  1271
+    name = this->name->toChars();
  1272
+    fd = open(name, flags, 0644);
  1273
+    if (fd == -1)
  1274
+        goto err;
  1275
+
  1276
+    numwritten = ::write(fd, buffer, len);
  1277
+    if (len != numwritten)
  1278
+        goto err2;
  1279
+
  1280
+    if (close(fd) == -1)
  1281
+        goto err;
  1282
+
  1283
+    if (touchtime)
  1284
+    {   struct utimbuf ubuf;
  1285
+
  1286
+        ubuf.actime = ((struct stat *)touchtime)->st_atime;
  1287
+        ubuf.modtime = ((struct stat *)touchtime)->st_mtime;
  1288
+        if (utime(name, &ubuf))
  1289
+            goto err;
  1290
+    }
  1291
+    return 0;
  1292
+
  1293
+err2:
  1294
+    close(fd);
  1295
+    ::remove(name);
  1296
+err:
  1297
+    return 1;
  1298
+}
  1299
+#endif
  1300
+
1283 1301
 /**************************************
1284 1302
  */
1285 1303
 
@@ -1306,7 +1324,7 @@ void File::writev()
1306 1324
 
1307 1325
 void File::appendv()
1308 1326
 {
1309  
-    if (write())
  1327
+    if (append())
1310 1328
         error("Error appending to file '%s'",name->toChars());
1311 1329
 }
1312 1330
 
7  src/root/root.h
@@ -162,6 +162,7 @@ struct File : Object
162 162
 
163 163
     void mmreadv();
164 164
 
  165
+
165 166
     /* Write file, return !=0 if error
166 167
      */
167 168
 
@@ -196,6 +197,12 @@ struct File : Object
196 197
      *  2:      directory
197 198
      */
198 199
 
  200
+#if POSIX
  201
+    /* POSIX implementation of write and append
  202
+     */
  203
+    int writePosix(bool append);
  204
+#endif
  205
+
199 206
     int exists();
200 207
 
201 208
     /* Given wildcard filespec, return an array of
9  src/statement.c
@@ -29,6 +29,7 @@
29 29
 #include "template.h"
30 30
 #include "attrib.h"
31 31
 #include "import.h"
  32
+#include "module.h"
32 33
 
33 34
 extern int os_critsecsize32();
34 35
 extern int os_critsecsize64();
@@ -439,7 +440,11 @@ Statements *CompileStatement::flatten(Scope *sc)
439 440
     }
440 441
     se = se->toUTF8(sc);
441 442
     Parser p(sc->module, (unsigned char *)se->string, se->len, 0);
442  
-    p.loc = loc;
  443
+    if (global.params.doMxnGeneration)
  444
+        p.loc = sc->module->importedFrom->writeMixin((unsigned char *)se->string, se->len, loc);
  445
+    else
  446
+        p.loc = loc;
  447
+    const size_t linstart = p.loc.linnum;
443 448
     p.nextToken();
444 449
 
445 450
     Statements *a = new Statements();
@@ -449,6 +454,8 @@ Statements *CompileStatement::flatten(Scope *sc)
449 454
         if (s)                  // if no parsing errors
450 455
             a->push(s);
451 456
     }
  457
+    if (global.params.doMxnGeneration)
  458
+        sc->module->importedFrom->mxnloc.linnum += p.loc.linnum - linstart;
452 459
     return a;
453 460
 }
454 461
 
9  test/compilable/extra-files/mixin-postscript.sh
... ...
@@ -0,0 +1,9 @@
  1
+#!/usr/bin/env bash
  2
+
  3
+diff --strip-trailing-cr compilable/extra-files/mixin.mixin ${RESULTS_DIR}/compilable/mixin.mixin
  4
+if [ $? -ne 0 ]; then
  5
+    exit 1;
  6
+fi
  7
+
  8
+rm ${RESULTS_DIR}/compilable/mixin.mixin
  9
+
37  test/compilable/extra-files/mixin.mixin
... ...
@@ -0,0 +1,37 @@
  1
+// mixin at compilable/mixin.d(64)
  2
+
  3
+int bar2()
  4
+{
  5
+    mixin Foo2;
  6
+    return result();
  7
+}
  8
+
  9
+// mixin at compilable/mixin.d(9)
  10
+5 + 3
  11
+// mixin at compilable/mixin.d(10)
  12
+int a1 = 5;
  13
+int b1 = 10;
  14
+auto res1 = a1 * b1;
  15
+
  16
+// mixin at compilable/mixin.d(16)
  17
+
  18
+          int a2 = 5;
  19
+          int b2 = 10;
  20
+          auto res2 = a2 * b2;
  21
+
  22
+// mixin at compilable/mixin.d(41)
  23
+5 + 3
  24
+// mixin at compilable/mixin.d(42)
  25
+int s1a = 5;
  26
+int s1b = 10;
  27
+auto s1res = s1a * s1b;
  28
+
  29
+// mixin at compilable/mixin.d(61)
  30
+
  31
+    int result()
  32
+    {
  33
+        auto tmp = a + mixin(bar2nested2());
  34
+        return tmp;
  35
+    };
  36
+// mixin at test_results/compilable/mixin.mixin(33)
  37
+getb
96  test/compilable/mixin.d
... ...
@@ -0,0 +1,96 @@
  1
+// PERMUTE_ARGS:
  2
+// REQUIRED_ARGS: -M -Mdtest_results/compilable
  3
+// POST_SCRIPT: compilable/extra-files/mixin-postscript.sh
  4
+
  5
+module foo.bar;
  6
+
  7
+void foo()
  8
+{
  9
+    auto a = mixin("5 + 3");
  10
+    mixin(
  11
+        "int a1 = 5;\n"
  12
+        "int b1 = 10;\n"
  13
+        "auto res1 = a1 * b1;\n"
  14
+    );
  15
+
  16
+    mixin("
  17
+          int a2 = 5;
  18
+          int b2 = 10;
  19
+          auto res2 = a2 * b2;
  20
+");
  21
+}
  22
+
  23
+string exp1()
  24
+{
  25
+    return "5 + 3";
  26
+}
  27
+
  28
+string stmt1()
  29
+{
  30
+    string s = "int s1a = 5;\n";
  31
+    s ~= "int s1b = 10;\n";
  32
+    s ~= "auto s1res = s1a * s1b;\n";
  33
+    return s;
  34
+}
  35
+
  36
+enum e1 = exp1();
  37
+enum s1 = stmt1();
  38
+
  39
+void bar()
  40
+{
  41
+    auto e1res = mixin(e1);
  42
+    mixin(s1);
  43
+}
  44
+
  45
+// Only generate output for string mixins
  46
+mixin template Foo()
  47
+{
  48
+    int x = 5;
  49
+}
  50
+
  51
+mixin Foo;
  52
+
  53
+struct Bar {
  54
+    mixin Foo;
  55
+}
  56
+
  57
+mixin template Foo2()
  58
+{
  59
+    int a = 5;
  60
+    int b = 3;
  61
+    mixin(bar2nested());
  62
+}
  63
+
  64
+mixin("
  65
+int bar2()
  66
+{
  67
+    mixin Foo2;
  68
+    return result();
  69
+}
  70
+");
  71
+
  72
+string bar2nested()
  73
+{
  74
+    return "
  75
+    int result()
  76
+    {
  77
+        auto tmp = a + mixin(bar2nested2());
  78
+        return tmp;
  79
+    };";
  80
+}
  81
+
  82
+string bar2nested2()
  83
+{
  84
+    return "getb";
  85
+}
  86
+
  87
+@property int getb()
  88
+{
  89
+    return 6;
  90
+}
  91
+
  92
+void main()
  93
+{
  94
+    bar();
  95
+    bar2();
  96
+}
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.