Comments: Projects' status & task list

[qwertie]

Associated with this page.

[jonathanvdc]

Hi! You may be interested in Flame.Loyc and fecs, a Loyc front-end for the Flame compiler framework, which is a managed compiler framework I've been working on for a while now.

Basically, Flame.Loyc is an F# library that converts Loyc syntax trees into Flame's (in-memory) IR. fecs is a C# driver program/AOT compiler that performs the following steps:

• make Loyc parse EC# or LES code (I haven't quite tested the latter, in all fairness, as I have so far concentrated my effort on EC# input);
• process the resulting syntax trees with LeMP;
• convert the processed syntax trees into IR;
• perform some static analysis of the IR, and issue warnings if questionable code is detected;
• (optionally) optimize the IR a bit;
• have the Flame middle-end and a back-end of your choice then gradually transform that IR into a CLR assembly or a number of C++/Python source code files (the C++/Python back-ends have their own runtime libraries, though: you can't just cross-compile a C# program that depends on the .Net framework classes).

Flame.Loyc's feature support is pretty basic at this point (no exciting stuff like generics, lambdas and events), but it can compile simple "hello, world"-like programs, such as this.

I'd love to hear your thoughts on this. Cheers!

[qwertie]

Wow, this Flame thing is really something else! Where's the documentation
of D#? What are your goals for Flame? It sounds an awful lot like Loyc!

On Fri, Aug 21, 2015 at 3:27 PM, Jonathan Van der Cruysse <
notifications@github.com> wrote:

Hi! You may be interested in Flame.Loyc and fecs
https://github.com/jonathanvdc/Flame.Loyc/, a Loyc front-end for the Flame
compiler framework https://github.com/jonathanvdc/Flame/, a managed
compiler framework I've been working on for a while now.

Basically, Flame.Loyc is an F# library that converts Loyc syntax trees
into Flame's (in-memory) IR. fecs is a C# driver program/AOT compiler
that performs the following steps:

• make Loyc parse EC# or LES code (I haven't quite tested the latter,
  in all fairness, as I have so far concentrated my effort on EC# input);
• process the resulting syntax trees with LeMP;
• convert the processed syntax trees into IR;
• perform some static analysis of the IR, and issue warnings if
  questionable code is detected;
• (optionally) optimize the IR a bit;
• have the Flame middle-end and a back-end of your choice then
  gradually transform that IR into a CLR assembly or a number of C++/Python
  source code files (the C++/Python back-ends have their own runtime
  libraries, though: you can't just cross-compile a C# program that depends
  on the .Net framework classes).

Flame.Loyc's feature support is pretty basic at this point (no exciting
stuff like generics, lambdas and events), but it can compile simple "hello,
world"-like programs, such as this
https://github.com/jonathanvdc/Flame.Loyc/blob/master/fecs/tests/Product.cs.

I'd love to hear your thoughts on this. Cheers!

—
Reply to this email directly or view it on GitHub
#3 (comment).

• David
  http://qism.blogspot.com

[jonathanvdc]

Well, yeah. Flame is kind of similar to Loyc. They're both managed, high-level compiler-ish frameworks.
Flame, however, deals with semantics most of the time, whereas Loyc seems to have a pretty clear focus on syntax: an LNode is really good at representing a syntax node, but to me it seems like an LNode's semantics are defined solely by the source language the syntax tree was parsed from, rather than by some intrinsic property of the node itself.
Flame's IR, the IExpression/IStatement interfaces - why yes, Flame does make a distinction between expressions and statements, though statements can be embedded in an expression and vice-versa -, on the other hand, would have absolutely no idea what to do when asked to print themselves: they only have well-defined semantics (because of their INode.Emit implementations), which can be leveraged to - among other things - generate C++/Python source code. Said generated code would then correspond to the input's semantics, but not its syntax.

My goal with Flame is to do for high-level languages what LLVM does for low-level languages. Unlike LLVM, which seems almost eager to discard non-essential information as soon as possible, Flame does not lower high-level constructs such as foreach, lambdas and yield unless told to do so explicitly. Flame's IR also uses structured control flow instead of labels and branches. Optimization and helpful diagnostics are important to me, too, though I find myself trying to get Flame's architecture and feature support right more often than I write optimization passes. World domination is something I'm slowly working toward, as well.

Which brings us to the awkward issue of the currently nonexistent D# docs. D#'s basically a dialect of C# that started off as a subset of C#, which then slowly expanded its syntax and semantics until it was a subset no more - though it does retain a lot of similarity to C#. I know I should get around to documenting D# someday, but I feel that's a pretty daunting task which will take up a lot of time that would be better spent squashing bugs and improving Flame's middle-end. Maybe I should write a short tutorial or comparison with C# that covers the relevant bits instead?

[qwertie]

It sounds like our goals are pretty much the same. Most of the Loyc code is currently focused on syntax because I guess I kind of felt like that was the "beginning", and I like to start things at the beginning. I haven't been able to find funding or other developers to help, so I haven't been able to flesh out other ideas. You're only the second person to contact me about Loyc - the first one wants to help, but has a full time job and little time to spare.

As you can see on the home page, I have a rudimentary plan for the semantic parts - a Standard Imperative Language (not yet designed), a Multi-Language Standard Library (not yet started).... and Enhanced C# which sounds a lot like D# - EC# is a superset of C#, but like D# it diverges quite a bit from C#, and it breaks backward compatibility in small ways.

I am unusual in that I tend to write my documentation before I write the implementation. Thus I've written multiple articles and documents about EC#, I just haven't published them "formally" yet. In particular, have a look at my "EC# for language pundits.txt" (download raw, since GitHub won't word wrap). The rules of EC# have changed slightly since I wrote the document... remind me to review it sometime. And since none of the compiler has been written except LeMP, most of EC# is subject to change. Even LeMP isn't really complete: I need a solid plan for allowing multiple macros to affect the same construct at the same time, e.g. imagine two attributes attached to a method that cause macros to insert different things into the method body. LeMP doesn't currently support that.

BTW I recently finished a pattern-matching macro for Loyc trees, which you'll probably find useful. See the unit tests for examples. I also made a macro for constructing a sort of "object-oriented" algebraic data type (examples in unit tests). How did you find out about Loyc? The first LeMP article?

Anyway, perhaps there is some way we could join forces? Could EC# and D# be combined into one language? Without good documentation of D# (and Flame), I can't guess. So I hope you'll read my unpublished article about EC#, then compare & contrast them for me.

[qwertie]

BTW a lot of the things "EC# for language pundits" says about macros are wrong, e.g. the syntax for defining macros has changed, there is not currently any macro hygiene system, it's called quote and not s_quote, LNode and not Node, etc. I really must revise this... it was written even before I made LES!

[jonathanvdc]

I've read up on EC# as you suggested. The way D# has diverged from C# mostly seems like a small subset of the features EC# has added. I'll try to break down the relevant points of convergence and divergence between D# and EC# here.

A (maybe not so) brief comparison between D# and EC

Things that D# and EC# agree on:

• D# and EC# both seem to like the D-style public this() { ... } as constructor syntax, though D# does not permit C#-style public Foo() { ... } constructors.
• Annotating parameters with set, e.g. public this(set int Value) { ... }. Same semantics as the macro in EC# (though D# does not require a using LeMP; declaration for set to work).
• const as a keyword that indicates function purity. Flame uses this as a hint when optimizing. Though nonstandard, the EC# parser generously accepts const, and Flame.Loyc reduces it to Flame's PrimitiveAttributes.ConstantAttribute.
• The virtual entirety of C#.

D#-specific stuff that C#/EC# don't do:

• Extension properties. Constructs such as public static bool IsAbstract[this IMember Member] { const get { ... } } are commonplace in Flame, and are consumed same as instance properties.
• Methods, constructors and accessors take a single statement as their body in D#, rather than a block statement like C# does. This is particularly useful when writing constructors, where usage of set has drained the constructor's body into oblivion, and writing an empty set of parentheses would otherwise be mandatory. With this, public Foo(int x, int y) { this.X = x; this.Y = y } becomes public this(set int X, set int Y);. Note the semicolon, which is a perfectly legal function body.
• static means "make this type a singleton" or "add this member to the enclosing type's associated singleton" (except when defining extensions - this is a C# extensions ABI compatibility problem). Writing out a singleton's type name is then made equivalent to accessing that singleton's instance. Flame's EmptyStatement is a fine example of this. Perhaps an EC# macro could do the same?

C#/EC# stuff that D# doesn't have:

• The typeof operator. D# tries to remain somewhat platform-agnostic, and as such does not yet have language-level support for reflection. I chose not to duplicate C#'s typeof operator because typeof(T) yields a System.Type, which would make D# reliant on the .Net framework forever.
• Generic type inference. This is something I just haven't got around to implementing yet.
• Macros in general.

Things that D# and C#/EC# do differently:

• C#'s continue; is next; in D#. I've always considered continue to be a particularly poor way of stating "jump to the next iteration", and next was my own sad attempt at rectifying the situation.
• D# handles delegate declarations differently. Instead of C#'s Func<int, double>, D# prefers double(int) - though Func<int, double> is by no means illegal -, which the CLR back-end then lowers to ye olde Func<int, double>. If I ever get around to implementing delegates in the C++ back-end, double(int) will likely be transformed into std::function<double(int)>, and the D# front-end will have no idea what Func<int, double> means, because Func<,> is part of the .Net framework core library.
• D# lambdas are slightly different from C# lambdas:
  • They are explicitly typed. x => x + 1 is int(int x) => x + 1 in D#.
  • Rather than replicating C#'s error-prone by-reference closures, lambdas capture by-value in D#.
  • Lambdas are a fairly new addition to the D# front-end and Flame in general. As a result, dsc's lambda support is full of bugs, which have taken up residence mainly in the front-end.
• ref has been replaced by T^ "reference pointers", i.e. pointers which don't do pointer arithmetic and other things that tend to cause segfaults. Pointer indirection on these T^ types is explicit. My reasoning for this is twofold:
  • ref causes hidden pointer indirection, which may conflict with other language features, such as lambdas.
  • T^ lets you know what's really going on. Sure, using pointers can be challenging, but if someone's not comfortable with using the pointer indirection operator (*) because they don't know what will happen if they do, they probably shouldn't be using ref either, because that merely hides the superficial syntax of pointer indirection, rather than the underlying semantics.

Template expansion

I'll elaborate on this because it's more of a philosophical topic to me.

Expanding templates is easy, but I would much rather tackle the few remaining template expansion use-cases with generics and hidden delegate parameters. This may sound like a performance killer, but that need not be the case if the generated code is sufficiently optimized.

For example, consider this D# snippet:

public const int Apply(int(int, int) op, int lhs, int rhs)
{
    return op(lhs, rhs);
}

public const int Plus(int lhs, int rhs)
{
    return lhs + rhs;
}

public const int ApplyPlus(int lhs, int rhs)
{
    return Apply(Plus, lhs, rhs);
}

Compiling this with -Og (dsc's default optimization level) will result in the following IL for ApplyPlus:

IL_0000: ldarg.0
IL_0001: ldarg.0
IL_0002: ldftn instance int32 Sugar.Program::Plus(int32, int32)
IL_0008: newobj instance void class [mscorlib]System.Func`3<int32, int32, int32>::.ctor(object, native int)
IL_000d: ldarg.1
IL_000e: ldarg.2
IL_000f: call instance int32 Sugar.Program::Apply(class [mscorlib]System.Func`3<int32, int32, int32>, int32, int32)
IL_0014: ret

Obviously, that's a major performance killer. Now. let's try compiling with the (albeit experimental and maybe even dangerous) -O3 optimization level.

IL_0000: ldarg.1
IL_0001: ldarg.2
IL_0002: add
IL_0003: ret

Same source code, different set of optimization passes. -finline and -fpropagate-locals kicked in with that -O3 option, to be specific. These passes make a living out of cheapening abstractions such as this. A generic constraint such as this could be introduced:

public T Add<T>(T lhs, T rhs)
    where T operator+(T, T)
{
     return lhs + rhs;
}

which would then be lowered to:

public T Add<T>(T lhs, T rhs, T(T, T) op_plus)
{
    return op_plus(lhs, rhs);
}

And the optimizer would make calling this "performance disaster" a breeze.

Now, I don't think "the compiler will take care of it for me" a lot when I write code. I do, however, believe that this is one of those rare situations where the compiler itself can at first generate suboptimal code because it knows for sure that specific instantiations of said code sequence will be optimized in the release build.

Conclusion

D# and EC# seem highly similar to me, especially if the right macros are used. Unifying them doesn't look all that hard to me, really. I do wonder, though, if we wouldn't be better off considering how features such as non-nullable references will fit in first. Last time I checked, the Roslyn folks were still arguing over how to add those to C# without breaking the default(T) operator. Creating a new programming language gives us a clean slate, which I think is an opportunity we shouldn't waste.

In response to your question: actually, I stumbled upon Loyc in a Google search results page, though I can't quite recall the specifics of my search query.

I'm also actively interested in developing a cross-platform standard library which can then be linked (and LTO'ed) with a platform-specific "primops" library. However, I don't see that happening any time soon without some intermediate format - not quite unlike SIL, though I would require said format to have an efficient, easy-to-parse binary representation: parsing source code has always proven to be prohibitively expensive in my experience.

Also, this comment ended up being way longer than I intended. Sorry about that.

[qwertie]

Let's continue this discussion in a new thread: #12

[qwertie]

I'm reconsidering the reassignment of LoycCore to core.ecsharp.net. Although LoycCore is the core libraries for EC#, it seems like it's just as appropriate to name it after "Loyc", it's has been published on NuGet as LoycCore for a couple of years, and I don't have an alternate name in mind ... maybe I should switch back to core.loyc.net after all. Any thoughts @jonathanvdc?

[jonathanvdc]

I've never operated an actual website myself, so I'm not sure if I'm qualified to comment on these matters. That being said, from a user's perspective: I don't think the exact URL matters that much, as long as it's consistent. I don't think switching back to core.loyc.net will hurt right now. But bear in mind that broken links are a pain. So if you're going to change Loyc's domain name, then I recommend you do it sooner rather than later.

By the way, I've recently created a Flame.Front NuGet package. I couldn't find any Loyc.Ecs and LeMP packages, so I just included the Loyc libs in the Flame.Front package. Right now, I'm using my own Loyc fork (which includes BLT) to build those libs, which complicates things a bit. I reckon I could just move the BLT format to a different repository if you were to create a Loyc.Ecs package, though.

Oh, and I'm using AppVeyor CI to build and publish NuGet packages automatically whenever I push a git tag. Perhaps that makes sense for Loyc as well. I could help you with writing an appveyor.yml file that does just that, if it's something you're interested in having.

[qwertie]

Well, if you ever want to do a website on GitHub it's quite straightforward (and btw I have blogged about setting everything up.) Okay, will change URL back to core.loyc.net. See #33 for the rest.