css

highlighter

canvasWidth

favicon

download

exportFilename

lineNumbers

fonts

unocss

shiki

850

/favicon.ico

true

slides

true

mono
Source Code Pro Bold

Migrating TypeScript to Modules

The Fine Details

Jake Bailey

Senior Software Engineer, TypeScript @ Microsoft

jakebailey.dev/talk-ts-congress-2023

What are we talking about?

More details at jakebailey.dev/go/module-migration-blog

An outline

What even is a "migration to modules"?
Why was it so challenging?
How did I make it less painful?
How did the migration actually work under the hood?
How did it go and what's next?

What even are modules?

A few different definitions... two most critical are:

Modules are a syntax (import, export)
Modules are an output format (ESM, CommonJS, SystemJS, AMD, UMD, IIFE, ...)

// @filename: src/someFile.ts
export function sayHello(name: string) { // Export from one file...
    console.log(`Hello, ${name}!`);
}

// @filename: src/index.ts
import { sayHello } from "./someFile"; // ... import it in another.

sayHello("TypeScript Congress");

TypeScript pre-modules

The opposite of modules is... scripts 😱 Everything is placed within global namespaces.

// @filename: src/compiler/parser.ts
namespace ts {
    export function createSourceFile(sourceText: string): SourceFile {/* ... */}
}

// @filename: src/compiler/program.ts
namespace ts {
    export function createProgram(): Program {
        const sourceFile = createSourceFile(text);
}

Fun fact: namespaces were originally called "internal modules".

Emitting namespaces

Namespaces turn into plain objects and functions.

// was: src/compiler/parser.ts
var ts;
(function(ts) {
    function createSourceFile(sourceText) {/* ... */}
    ts.createSourceFile = createSourceFile;
})(ts || (ts = {}));

// was: src/compiler/program.ts
var ts;
(function(ts) {
    function createProgram() {
        const sourceFile = ts.createSourceFile(text);
    }
    ts.createProgram = createProgram;
})(ts || (ts = {}));

"Bundling" with `prepend`

// @filename: src/tsc/tsconfig.json
{
    "compilerOptions": { "outFile": "../../built/local/tsc.js" },
    "references": [
        { "path": "../compiler", "prepend": true },
        { "path": "../executeCommandLine", "prepend": true }
    ]
}

Makes tsc emit:

var ts;
// Cram all of src/compiler/**/*.ts and src/executeCommandLine/**/*.ts on top.
(function(ts) {/*...*/})(ts || (ts = {}));
// ...
// was: src/tsc/tsc.ts
(function(ts) { ts.executeCommandLine(...); })(ts || (ts = {}));

What if someone wants to import us?

Our outputs are constructed global scripts, but we can be clever.

namespace ts {
    if (typeof module !== "undefined" && module.exports) module.exports = ts;
}

Emits like:

var ts;
(function(ts) {/* ... */})(ts || (ts = {}));
// ...
(function(ts) {
    if (typeof module !== "undefined" && module.exports) module.exports = ts;
})(ts || (ts = {}));

Namespaces have some upsides

With namespaces, we don't have to write imports, ever!

When adding code, no new imports
When moving code, no changed imports
tsc "bundles" our code using prepend

But...

Nobody writes code like this anymore!

We don't get to dogfood modules
We can't use external tools
We have to maintain prepend... but nobody uses it except us 🥴

What we want:

// @filename: src/compiler/parser.ts
export function createSourceFile(sourceText: string): SourceFile {/* ... */}

// @filename: src/compiler/program.ts
import { createSourceFile } from "./parser";

export function createProgram(): Program {
    const sourceFile = createSourceFile(text);
}

We know what we want; let's do it

The question is... how can we:

Actually make the switch ...
... while maintaining the same behavior ...
... and preserving a compatible API?

layout: center

The challenge

TypeScript is huge!

TypeScript changes often!

That's an average of ~5 commits a weekday.

How can we change a huge, moving project?

Certainly not by hand! Automate everything.

Code transformation where possible
git patches to store manual changes
Done stepwise, for debugging, review, git blame preservation

What does the migration tool look like?

Code transformation is performed with ts-morph
- An extremely helpful TypeScript API wrapper by David Sherret ❤️ (ts-morph.com)
Manual changes are managed by git with .patch files!
- git format-patch dumps commits to disk
- git am applies the patches during the migration
- If a patch fails to apply, git pauses for us!

jakebailey.dev/go/module-migration-demo

layout: center

Code transformation

Step 1: Unindent

Eventually, we will pull the code out of the namespaces, one block higher.

If we do it early, later diffs will be cleaner, and git will remember.

From:

namespace ts {
    export function createSourceFile(sourceText: string): SourceFile {/* ... */}
}

Into:

namespace ts {
export function createSourceFile(sourceText: string): SourceFile {/* ... */}
}

Step 2: Make namespace accesses explicit

Namespace accesses are implicit, but imports will be explicit.

From:

export function createSourceFile(sourceText: string): SourceFile {
    const scanner = createScanner(sourceText);
}

Into:

export function createSourceFile(sourceText: string): ts.SourceFile {
    const scanner = ts.createScanner(sourceText);
}

This will make the next step clearer.

Step 3: Replace namespaces with imports

Given:

namespace ts {
export function createSourceFile(sourceText: string): ts.SourceFile {
    const scanner = ts.createScanner(sourceText);
}
}

We'll convert this into:

import * as ts from "./_namespaces/ts";

export function createSourceFile(sourceText: string): ts.SourceFile {
    const scanner = ts.createScanner(sourceText);
}

Step 3: Replace namespaces with imports

As a diff:

-namespace ts {
+import * as ts from "./_namespaces/ts";
+
 export function createSourceFile(sourceText: string): ts.SourceFile {
     const scanner = ts.createScanner(sourceText);
 }
-}

Everything inside is unchanged!

But, what the heck is this _namespaces import?

Introducing... "namespace barrels"

Using reexports, we can build modules whose exports match the API of the old namespaces.

// @filename: src/compiler/_namespaces/ts.ts
export * from "../core";
export * from "../corePublic";
export * from "../debug";
// ...

// @filename: src/compiler/checker.ts
import * as ts from "./_namespaces/ts";

// Use `ts` exactly like the old namespace!
const id = ts.factory.createIdentifier("foo");

These are often referred to in the JS community as "barrel modules".

For us, these become our namespaces, so let's just call them "namespace barrels"!

Emulating namespace behaviors

Most namespace behavior can be emulated with modules.

// Emulate nested namespaces like `namespace ts.performance {}`
// @filename: src/compiler/_namespaces/ts.ts
export * as performance from "./ts.performance";

// Emulate `prepend` by reexporting multiple namespace barrels
// @filename: src/typescript/_namespaces/ts.ts
export * from "../../compiler/_namespaces/ts";
export * from "../../services/_namespaces/ts";
export * from "../../deprecatedCompat/_namespaces/ts";

// Export the entire ts namespace for public use
// @filename: src/typescript/typescript.ts
import * as ts from "./_namespaces/ts";
export = ts;

Step 4: Convert to named imports

After step 3, we're left with namespace imports, like:

import * as ts from "./_namespaces/ts";

export function createSourceFile(sourceText: string): ts.SourceFile {
    const scanner = ts.createScanner(sourceText);
}

This step converts them to named imports:

import { createScanner, SourceFile } from "./_namespaces/ts";

export function createSourceFile(sourceText: string): SourceFile {
    const scanner = createScanner(sourceText);
}

The tedious work is done!

At this point, we're done with the bulk transformation.

But, we're not all the way there yet.

Manual changes

After the automated transform steps, there are 29 manual changes!
This is obviously scary; any changes to main could conflict
But, we are using git to manage these!
If we run the migration, git will pause, just like a rebase. Just:
1. Fix the problem
2. git am --continue
3. Ask the migration tool to dump the patches

layout: center

Some manual change highlights

Bundling with `esbuild`

We still needed to bundle
Lots of bundlers to choose from; I went with esbuild (esbuild.github.io)
Obviously, it's fast; ~200 ms to build tsc.js
Features scope hoisting, tree shaking, enum inlining
We maintain a mode in our build which uses solely tsc, just to be sure

`d.ts` bundling

Without tsc's prepend, someone needs to bundle d.ts files
I ended up rolling my own d.ts bundler (~400 LoC)
Definitely not for external use; it's very specific to our API

// Something like...
namespace ts {
    function createSourceFile(): SourceFile;

    namespace server {
        namespace protocol {
            // ...
        }
    }
}
export = ts;

Complete build overhaul

Our old build was handled gulp; had gotten somewhat convoluted and hard to change
With modules, the build steps are quite different!
Build completely replaced, reimplemented using an all new task runner (~500 LoC)
It's called hereby, don't use it, thanks 😅

export const buildSrc = task({
    name: "build-src",
    description: "Builds the src project (all code)",
    dependencies: [generateDiagnostics],
    run: () => buildProject("src"),
});

We did it! How has it turned out?

Great! 👍

For TypeScript users:

10-20% speedup from esbuild's scope hoisting
43% package size reduction (63.8 MB -> 37.4 MB)
No API change

For the TypeScript team:

Core development loop improvement
Dogfooding!
prepend is deprecated; to be removed in TS 5.5

What's next?

There's way too much exciting stuff to talk about, but:

Getting rid of _namespaces, somehow?
Shipping ESM?
- Probably works for executables (tsc, tsserver, ...)?
- Maybe an ESM API "for free"?
Untangling things so we can be tree shaken?
- Could we have @typescript/* packages?
Minification? Other optimizers?
- Downstream patchers make this challenging 😢

Thanks!

Find me at: jakebailey.dev

These slides: jakebailey.dev/talk-ts-congress-2023

The migration PR: jakebailey.dev/go/module-migration-pr

The migration tool: jakebailey.dev/go/module-migration-tool

Module migration blog: jakebailey.dev/go/module-migration-blog

Watch the migration in real time: jakebailey.dev/go/module-migration-demo

Files

slides.md

Latest commit

History

slides.md

File metadata and controls

Migrating TypeScript to Modules

The Fine Details

Jake Bailey

Senior Software Engineer, TypeScript @ Microsoft

What are we talking about?

An outline

What even are modules?

TypeScript pre-modules

Emitting namespaces

"Bundling" with prepend

What if someone wants to import us?

Namespaces have some upsides

Nobody writes code like this anymore!

We know what we want; let's do it

layout: center

The challenge

TypeScript is huge!

TypeScript changes often!

How can we change a huge, moving project?

What does the migration tool look like?

layout: center

Code transformation

Step 1: Unindent

Step 2: Make namespace accesses explicit

Step 3: Replace namespaces with imports

Step 3: Replace namespaces with imports

Introducing... "namespace barrels"

Emulating namespace behaviors

Step 4: Convert to named imports

The tedious work is done!

Manual changes

layout: center

Some manual change highlights

Bundling with esbuild

d.ts bundling

Complete build overhaul

We did it! How has it turned out?

What's next?

Thanks!

"Bundling" with `prepend`

Bundling with `esbuild`

`d.ts` bundling