Allow use with type annotations?

[seanhess]

I've been watching Sweet.js for a while. I love the project. Especially, I agree with the idea that our compile-to-js transformations should be composable.

I've been a fan of strict types for a while, and right now type systems conflict with sweet's goals. A type system not only has to parse all the code, it has to understand all the code. This makes composable macros difficult, because the type system can't possibly know all the crazy things people will do with them.

But the macro system itself doesn't have to understand all the code.

Could we change sweet.js to allow the use of type annotations? I'm not saying it should understand them, or do anything with them, but it shouldn't crash if it doesn't understand them. Today, if I try to run sweet on a Typescript file, it crashes because it doesn't have a macro for the annotations.

Today is it possible to define a pass-through for all Typescript-esque type annotations? The idea would be run run sweet.js first, then run the type checker on the resulting type-annotated code. If not could we change sweet so it doesn't die if it doesn't understand something?

It's ironic that today sweet.js allows you to compose macros and transformations, but it isn't itself composable with any other transformation system.

I think this is especially relevant given the announcement of Flow from facebook, and their choice to adopt Typescript-like annotations. My prediction is that developers will start to gravitate towards strict types within the next 3 years, and I would hate to see people have to choose between type safety and macros.

Thoughts? I'm happy to help if I can. I have significant time to dedicate to making cool stuff like this happen.

[natefaubion]

The main issue is that sweet.js has to take in JS, or macro-ed JS that becomes vanilla JS. It can't take in something that looks kind of like JS, but with this or that being different, because then it can't understand the structure of your program so that it can apply hygiene. Take for example a typed declaration like this:

var foo: Foo = Foo(12)

This is not a normal var declaration. This would be a parse error in JS. The expander needs to be able to parse var declarations so it can track scope and hygiene. If we just let this "pass through" then we would never know that foo is supposed to be a var in scope (which is very bad). OK, so say we patch sweet.js so that it understands this form. Now someone else comes along with a slightly different type system that looks like this:

var Foo foo = Foo(12)

Which is totally different from the original one. What do we do? Do we keep patching sweet.js until it understands all these forms? What if someone tried to mix and match forms? That doesn't really make sense.

But what if we could extend that macro expansion process ourselves on a module by module basis? A type system as a macro? That's basically what Typed Racket is. Racket lets you declare your module as a specific "language", wherein you can hand over your entire module to a big macro. This macro can analyze your syntax and do fancy stuff like typechecking.

So the answer to integrating type systems and sweet.js isn't so much that the output of the compiler needs to feed into the input of another, but that you can call out to other type systems as libraries while expanding your modules. There is still quite a bit of work do be done to this end.

---

Additionally, sweet.js has to parse the output as JS so that it knows what you've expanded isn't complete nonsense. Otherwise you wouldn't know your expanded program isn't full of errors. We also pretty print it from the AST using escodegen. I suppose you could technically pretty print it from raw tokens if you had such a printer.

[seanhess]

Interesting. I wasn't thinking about hygiene. That makes sense.

It seems like the type system guys are moving so fast and they aren't thinking about making this all work. Does it require full cooperation from the designers of the type checker, or a type system built from the ground up to work this way? Will it be possible to do cross-module checking with this approach?

It seems like the Facebook guys are more likely to listen to feedback at this point, and that they are more likely to influence the direction of JS type checking in the future. If making this all work requires cooperation from the designers of the type system, maybe we should get in contact with them right away?

[natefaubion]

Yes, you could do cross-module checking with it. With module support, macros will be able to keep state, so there's no reason you couldn't maintain a table of externs.

Anything would require a significant amount of work in sweet.js land to get the syntax into a form your typechecker library would understand. For example, maybe there's one primary function exported called typecheck(ast: TypeScriptAST, externs: ModuleExterns). Your language macro would need to get the module's syntax into an AST the typechecker can understand, and provide the externs for any imported modules.

Also keep in mind that typechecking would only be performed on modules that opt in using the appropriate module-level declaration. If there's no declaration, the macro is never invoked!

[BrendanEich]

Ecma TC39 has "types" on its agenda, see 5(iv) at https://github.com/tc39/agendas/blob/master/2014/09.md, but we won't make the mistake we made in ES4 of rushing. This is why we are not standardizing TypeScript, and we're eager to learn more about Flow from Facebook folks on TC39.

Ideally we will have co-evolution of sweet.js and JS post-ES6 (specifically, JS with modules) until the TypedRacket approach comes alive. That would be sweet!

/be

[andreypopp]

FWIW Facebook's fork of esprima already can parse type annotations. Don't know if it's difficult or not to adapt it to use with sweet.js instead of esprima.

[backspaces]

Is there a Sweet tutorial/example of making Types/TypedArrays easier to use .. and hopefully with serious improvement in both cpu performance and memory efficiency? Sorta a struct package?

I ask because one of our projects recently moved from an Array of Objects to an Object of Typed Arrays, a different TA for each variable that would ordinarily be an object property.

So instead of an array of many objects like {x:0, y:1} we have a single object {x:Uint16Array, y:Uint16Array}. This may be extreme, but it actually has turned out to be not too bad to use. Its way faster, webgl friendly, and much Much more memory efficient.

But we'd like to be even better with a syntax more like es7 might have. It currently is a bit difficult for new devs and a little Sugar would be Sweet!

[seanhess]

Super cool. I'm glad you guys are talking about types at the next meeting @BrendanEich!

I'm starting a new project for a startup and we're trying to make some tech decisions, but we're caught between types and everything else (generators, JSX, sweet, everything). My motivations are practical: I want to figure this out for my project.

I want to work on something to help resolve the issue in the short-term if possible, while still taking the long-term view of the standardization of everything. Is there something practical and useful that could be done soon, that doesn't conflict with the long term? I can learn whatever I need to learn.

[disnet]

@backspaces I seem to remember @jlongster had hacked up some macros for TypedArrays a while ago. I think it was struct.js.

[elibarzilay]

I know that this is old, but FWIW, re the Typed Racket mentions of @natefaubion, @BrendanEich:

It's wrong to say that a macro system should be tied to a type system -- and Typed Racket is a good example of that. Yes, it's one big macro that does typechecking (and optimizations, even), but the very first step that it's doing is to completely expand the code so that it can typecheck just the core syntax.

Translated to sweet.js terms, it just needs to learn how to parse these JS-like languages and spit them out for the type tools to process. It's obviously more than just a plain addition of stuff that it ignores, since types have identifiers, but if Flow and TypeScript have the same syntax, then there is just one JS-like language to deal with, even if the semantics implemented by the two type systems are different.

A very different issue is allowing the typechecker to interact with macros -- like allowing macros that expand differently based on the current type. Something like this would be impossible with piping a checker behind a macro expander, but even Typed Racket doesn't have that. -- It would require that TR implements its own macro expansion, with the additional type information injected in there. (Or some way for the type checker to jump back to the macro generation, which I once tried via an ugly continuation hack...)