Emit dependencies and proper type names #46

joeduffy · 2017-01-11T16:12:44Z

At the moment, all of our conversions are "syntactic"; that is, with few exceptions, we only rely on the parse tree information to generate the MuPack/MuIL tree. This means we don't emit bound types from TypeScript's semantic analysis tree, hence some random "TODO" types in the output.

We need to start emitting fully bound information. This is more complex than it seems, because we not only need to consult the TypeChecker -- versus relying exclusively on the AST nodes as we are doing now -- and, perhaps more difficult, we must lower the type names to proper MuIL symbol tokens. These must be references as described in https://github.com/marapongo/mu/blob/master/docs/design/deps.md. As part of that, we will need to also emit the dependencies.

Emitting dependencies will also be difficult, since all dependencies in TypeScript are source-based, and so this information is "lost" by the time we are compiling. We may need to recover it from the package metadata in the locations TypeScript has picked up certain definition files from.

At the moment, we do not emit bound node information yet. (See #46). When that day comes we will need to decide whether import declarations are needed or, as I suspect, they can continue to be discarded safely because all dependency information is available on the bound nodes themselves. I'm leaving in a TODO in there just as a reminder to resolve this at that time.

TypeScript type aliases simply turn into MuIL classes with the base class being the aliased type. In theory, MuIL conversions will support the kind of conversions offered by aliases (or at least get sufficiently close). We will need to tackle this in part by #46. In addition, while I was in there, I cleaned up the type token emission to emit "any" (rather than "TODO") and to use a single, central function for all ts.TypeNodes. This will also get cleaned up by #46.

This change first and foremost adds a MuIL export node (ast.Export); this is for re-exporting members of other modules, potentially under a different name. This works by simply by associating a name with a fully qualified token name that, when binding, can be re-resolved to the real target. This should be sufficient for expressing everything we need from MuJS. In addition to that, there is an initial cut at basic re-exports; this is the form where you would say: export { some, nifty as cool, stuff } from "module"; In this example, we end up with two Export nodes: one maps "some" to "some/module", "cool" to "some/nifty", and "stuff" to "some/stuff". Note that we have not yet implemented two other variants. First, we will want to support the "export all" variety, e.g. export * from "module"; Instead of supporting wildcards in MuIL -- which would result in the odd attribute that you couldn't know the totality of a module's exports without also examining its dependencies (possibly recursively) -- we will require the compiler to map the "*" to a concrete list of Export nodes. Second, we also want to support named exports from the current module. That is to say, export clauses without `from "module"` parts. E.g. let some = ...; let nifty = ...; let stuff = ...; export { some, nifty as cool, stuff }; This is less important because you can always export members using the more familiar `export <decl>` syntax (which is already recognized by MuJS). But of course, we will want to support it eventually. It is slightly more complex, because we will need to use static typing to resolve the exported members to their types -- e.g., variable, class, interface -- and in fact this could lead us to re-exporting whole modules in their entirety. Both of these two will require #46. Since so much is piling up behind this, I will likely begin biting this off soon. I also believe that we may want re-export nodes, Export, to carry a specifier that indicates whether the target is a class, variable, or sub-module.

This change adds true understanding and dependency resolution for modules. This requires consulting the bound node tree and TypeChecker object associated with the TypeScript program AST. This introduces a so-called ModuleReference, which is just a wrapper around a TypeScript import string, enabling us to continuously look up information about a module's symbol (from the SourceFile map). It only lowers to a MuIL ModuleToken for serialization at which point all information required to load that dependency is expected to have been pre-arranged according to our package manager resolution process (see deps.md). As part of this, we can implement the `export * from "module"` export form. This is a huge step forward for #46.

This change emits more types. In particular: * Previously, only primitive types got emitted, yielding "any" for any custom types. Now we emit custom types, including fully qualified module names for type references resolving to imported modules. * Prior to this change, we erroneously used the type node on the function declaration itself as an approximation for return type. To get the true return type, we need to dig through a few nodes, including the Declaration and Signature. This change now properly emits return types. This doesn't close out #46, however we are getting close.

joeduffy · 2017-01-28T19:36:08Z

This is very close to working. However, the remaining work is to audit all tokens, and ensure they are fully qualified. We are still emitting identifier-style names in most places.

For example, the tests/output/modules/var_1 output contains this:

{
    "kind": "LoadLocationExpression",
    "name": {
        "kind": "Token",
        "tok": "x"
    }
}

when it should actually contain this:

{
    "kind": "LoadLocationExpression",
    "name": {
        "kind": "Token",
        "tok": "var_1:index/x"
    }
}

Notice the difference in the token: x (incorrect) versus var_1:index/x (correct).

This overhauls how MuJS creates and emits tokens, so that they are properly fully qualified. Before this change, we frequently emitted "naked" identifier names that, though the compiler had already resolved and bound them, looked to the MuIL toolchain as though they hadn't been. This required rearranging the "symbols" (now "tokens") layer to look more like the types in the MuIL toolchain. More impactfully, however, it means we need to load MuIL packages from dependencies as part of creating the fully resolved tokens. This injects I/O into places it didn't exist before, so a large part of this change is simply playing the async/await whack-a-mole game. This is one big step along the path of fixing #46. It still isn't 100% because we need to emit array/property symbols correctly, and also I suspect as we do more multi-module testing, a few other things will crop up. But we're almost there.

joeduffy · 2017-02-10T17:42:13Z

This is done, excepting for unexpected bugs, of course. So closing out.

joeduffy added area/tooling kind/enhancement Improvements or new features labels Jan 11, 2017

joeduffy added this to the S9 milestone Jan 11, 2017

joeduffy self-assigned this Jan 11, 2017

joeduffy mentioned this issue Jan 17, 2017

Initial MuJS implementation #40

Closed

joeduffy closed this as completed Feb 10, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Emit dependencies and proper type names #46

Emit dependencies and proper type names #46

joeduffy commented Jan 11, 2017

joeduffy commented Jan 28, 2017 •

edited

joeduffy commented Feb 10, 2017

Emit dependencies and proper type names #46

Emit dependencies and proper type names #46

Comments

joeduffy commented Jan 11, 2017

joeduffy commented Jan 28, 2017 • edited

joeduffy commented Feb 10, 2017

joeduffy commented Jan 28, 2017 •

edited