Questions about the compiler

[unintelligible]

Hi,

I came across the Saltarelle compiler through the SharpDevelop forums, when looking for info about NRefactory. I was writing my own JS compiler, and am 2/3 way done; however, looking through your code, it seems you are quite a bit further ahead than me. It seemed like a good idea to check what your plans are for this before continuing work on mine; it may be better not to duplicate effort?

I notice that the code doesn't yet handle end-to-end conversions ('output writer' is still in your todo list.) I assume that the difficulty here is figuring out the dependencies between compiled C# files, so that the source is printed out in the correct order (I'm also assuming that only the static initializers would need to be checked for dependencies?)

The key focus of my compiler was that it should be easy for users to plug their own functionality in - e.g. if people wanted to change how C# classes are implemented in JS, they could override it easily in the compiler; or if someone wanted to implement say parts of Mscorlib in the compiler (e.g. by transforming a List to an array and converting the List methods to JS code), or implement the async keyword, or implement LINQ support, then the compiler should support that. The idea was to provide a 'compiler framework' of sorts, that would handle the basics but be extensible. Does this sound interesting?

Cheers,
Nick

[erik-kallen]

I'm glad someone is interested in my work.

My plans with the compiler is to someday put it in a state where it can do a full transform of C# to JS. I do think, however, that this needs to be done in a bottom-up fashion, where all parts have to work and being thoroughly tested before gluing them together, since a compiler that works 95% is of 0% value.

I'm thinking the same way as you when it comes to extensibility, except that I don't think that it is feasible to allow suppport for eg. async to be pluggable.

The "output writer" you are referring to is the stage that is supposed to take "Objective JavaScript" classes and transform it to JavaScript statments, so the method signature would be something like:

IEnumerable<Saltarelle.Compiler.JSModel.Statements.JsStatement> Transform(IEnumerable<Saltarelle.Compiler.JSModel.TypeSystem.JsType> typesInAssembly)

I don't think classes have to be printed in any specific order, as long as it's cleverly done (Script# is clever about this). Dependencies between static initialization statements in different classes are problematic, but I don't think it's the end of the world if this is not supported in v1. The reason for it not being done is my bottom-up approach to writing the compiler.

Operations like transforming a list to an array are the responsibility of the metadata importer (a to-be-written implementation of the to-be-renamed-or-split interface Saltarelle.Compiler.INamingConventionResolver). This is intended to read attributes from classes and members, so if you want to transform a List to an Array, you'd define an mscorlib with a class looking something like this:

public class List<T> {
    [LiteralCode("[]")]
    public List() {
    }

    [ScriptName("push"), IgnoreGenericArguments]
    public void Add(T obj) {
    }
}

which would make code like

var l = new List<int>();
l.Add(1);

be compiled to

var l = [];
l.push(1);

The plan is to use the Script# mscorlib (but it needs a few improvements eg. because Script# does not support generics).

When it comes to LINQ support, I think NRefactory will handle query comprehension syntax (transform var x = from item in myList select new { x = item } to var x = myList.Select(item => new { x = item }) and getting it to actually work would mean to write an import library for eg. LinqJS or JSLinq.

If you want to help in some way, you are very welcome!

[unintelligible]

Hi,

thanks for the quick response. Would definitely be interested in helping; the code you have looks very well thought through.

I think the facility to generate a JS AST which doesn't necessarily correspond to the C# source could be quite important. For instance, both JSLinq and LinqJS require the source JS array to be wrapped and unwrapped before/after running LINQ methods on it:

IEnumerable.FromArray(myArray).Where(function(x) { return x + 1; }).ToArray()

It would be possible to implement as a special case in the compiler. Similarly, ScriptSharp has the ability to generate QUnit tests from C# code; something which can't be achieved simply by translating C# statements to the JS equivalent. I suspect this is also implemented as a special case in the compiler; however, it would be better if this could be implemented as a plug-in of sorts, so that adding support for a new JS construct doesn't require changing the compiler itself. I think this would make the compiler much more useful, because it wouldn't depend on the compiler maintainer adding support for different JS-isms - users could do that themselves by writing a compiler plugin.

The way I implemented this in my own compiler was to create a series of interfaces (e.g. IMemberReferenceExpressionCompiler, ITypeDeclarationCompiler) which classes can implement to handle specific C# constructs. The implementation has responsibility for transforming a C# construct to one or more JS statements/expressions. Different implementations of the interface can be registered with the compiler; the implementations are called in order, with each implementation deciding whether this is a construct it can handle - if not, the next implementation is called, until the default implementation (which throws a NotImplementedException) is reached.

Is this an approach you might be interested in (particularly for handling type and method declaration and references)?

Also, my main aim (with my own compiler) was to get something up and running fairly quickly, as I have an immediate requirement to use it. If I wrote an implementation for the OutputWriter (with the aim of moving towards a compiler that can generate JS, even if certain constructs are not supported), would you consider accepting a pull request?

[erik-kallen]

For the LINQ case, I think this could be handled in the metadata by defining classes like:

public static class Enumerable {
    [Implementation("JSLINQ({src}).Select({f})"]
    public static IWrappedEnumerable<TResult> Select<TSource, TResult>(IEnumerable<TSource> src, Func<TSource, TResult> f) {}

    [Implementation("JSLINQ({src}).Where({f})"]
    public static IWrappedEnumerable<TSource> Where<TSource>(IEnumerable<TSource> src, Func<TSource, bool> f) {}

    //... Also other methods...
}

public interface IWrappedEnumerable<T> : IEnumerable<T> {
    IWrappedEnumerable<TResult> Select<TResult>(Func<T, TResult> f) {}

    IWrappedEnumerable<TSource> Where<TSource>(Func<T, bool> f) {}
}

but I must admit to not having thought this through 100%.

As for QUnit, I don't know how Script# does it because I only use v0.5.5 (because I don't dare upgrade because you never know what stops working, and I aim to upgrade to my own compiler once that is finished), but I think currently an API could be based around code like

public void DoIt() {
    QUnit.Module("Module1");
    QUnit.Test(() => {
        Assert.Same("a", "a");
    });
}

Not perfect, but could work as an interim solution.

As for your interface-based approach, the problem is that you, in general, cannot compile an expression in isolation from other expressions. For example, given the code

class C {
    public int P1 { get; set; }
    public int P2 { get; set; }
    public int F() { return 0; }

    public void M() {
        P1 = P2 = F();
    }
}

the compilation of M() would look completely different depending on how the properties are implemented. In my compiler, I allow properties to be treated as fields, which gives the code

this.P1 = this.P2 = this.F();

but they could also use get/set methods, which would give the code

var $tmp1 = this.F();
this.set_P1($tmp1);
this.set_P2($tmp1);

(note how we need to introduce a temporary variable in order to not evaluate F() twice). Reordering of named arguments is also kind of hard.

The ITypeDeclarationCompiler interface sounds more promising, though, perhaps also a sibling IMethodCompiler.

Of course I will accept pull requests, as long as they have good test coverage and do things that I want to be done (I promise to check your fork and comment and discuss things).

Having the output writer (although it should probably be called something else) would be nice, but there are also other parts that are required before the compiler is actually usable (just so you don't get disappointed if it doesn't work after submitting your pull request):

• The metadata importer, which is an implementation of the INamingConventionResolver interface (that interface should be split/rename in the future because it does way too many different things). Preferrably, this module should have the ability to minimize non-public types and members.
• The reference importer, whose API is still to be determined, but the simplest implementation is trivial. The idea is that whenever a type reference is needed, the compiler just inserts a JsTypeReferenceExpression, and it is the responsibility of this module to transform that to some valid JavaScript (this is where support for AMD will be added in the future). Wouldn't take more than an hour to write the first implementation.
• The runtime library, which is an implementationt of the IRuntimeLibrary interface. Getting this to work will most likely involve forking ScriptSharp, and then performing necessary modifications of its mscorlib.
• A driver, whose purpose is to invoke all the other parts in the correct order. As a first step in this driver, I intend to send the source code through the copy of mcs that is embedded in NRefactory, to ensure that the code is valid C#.
• An exe and an msbuild task (perhaps a NAnt task as well?) which are intended to be thin wrappers around the driver.

[erik-kallen]

The compiler now works quite well. Help is still appreciated.

[unintelligible]

Hi,

I just took a look at this again over the week-end. The compiler looks
fantastic. I might be filing a couple of issues/suggestions over the
next few days; beyond that, what else do you need help with? I can't
guarantee any commitment I'm afraid (startup), but if I do get any
time I'm happy to help out.

Thanks,
Nick

On 28 June 2012 01:56, erik-kallen
reply@reply.github.com
wrote:

The compiler now works quite well. Help is still appreciated.

---

Reply to this email directly or view it on GitHub:
#1 (comment)