feat: typescript project reference incremental build support

[electroma]

I have a TypeScript monorepo setup with multiple node modules and TypeScript project references between them.
TSC does support tsc -b which will perform an incremental build on the module and all referenced projects.

Both rollup and webpack plugins have support for this feature.
Though I can't see anything like that in ESBuild.
Am I missing something?

[evanw]

Do you mean this? https://esbuild.github.io/api/#incremental

[electroma]

Incremental build on one module level is a different thing.
TypeScript project references work differently, since they allow to perform incremental build across multiple Node modules.
An example could be an "App" module referencing a "Library" node module.
When project references are configured in tsconfig.json, TS compiler may perform on-demand compilation of the referenced "library" module (and any transitive dependencies) on-demand.
It is especially useful in the context of monorepo, when multiple Node modules share a single GIT repository.

[nihalgonsalves]

@electroma Do you mean using TypeScript Project References + referencing a non-yet-built file in the main/module/exports in one of the package.json files?

There are two workarounds for this:

• Specify compilerOptions.paths, so esbuild knows where to look. This duplicates some configuration, since you'd be specifying the source paths for each package here, while specifying the built paths in the package.json.

• or set your main/module/exports field to the TS source file. (e.g instead of "main": "dist/foo.js" set "main": "src/foo.ts"). This works perfectly, but you cannot publish such a package since the main field is invalid outside the context of your monorepo.

---

@ Evan - the use case is something like this:

• package.json

  {
    // Using yarn (or npm) workspaces, the monorepo dependencies are symlinked.
    "workspaces": ["./pkg-a", "./pkg-b"]
  }

• tsconfig.json:

  "Solution" style. This file doesn't really matter for the example, but is usually used along with tsc --build to build all projects.

  {
    // includes no files itself
    "include": [],
    // references all tsconfigs
    "references": [
      { "path": "./pkg-a/tsconfig.json" },
      { "path": "./pkg-b/tsconfig.json" }
    ]
  }

• pkg-a/

  • package.json

    {
      "name": "pkg-a",
      "dependencies": {
        // depends on monorepo pkg-b
        "pkg-b": "*"
      }
    }

  • tsconfig.json

    {
      "include": "./src",
      // referenced here too
      "references": [{ "path": "../pkg-b/tsconfig.json" }]
    }

  • src/hello.ts

    // this is usually resolved to `dist/index.js` (see below, pkg-b/package.json -> main)
    // however, in a monorepo, these files are not always built, or if they are, are likely out of date.
    
    // the project reference above tells TypeScript to look at the tsconfig.json of `pkg-b/`,
    // and seamlessly build or update the built files if they are out of date
    
    import { x } from "pkg-b";
    
    console.log(x);

• pkg-b/

  • package.json

    {
      "name": "pkg-b",
      // resolved here
      "main": "dist/index.js"
    }

  • tsconfig.json

    {
      "include": "./src",
      "compilerOptions": {
        "rootDir": "./src",
        "outDir": "./dist"
      }
    }

  • src/index.ts

    export const x = "hi";

So: esbuild cannot simply follow the references and build the other project, since it wouldn't fit into how esbuild works.

Something that is theoretically possible would be to read the referenced tsconfig.json and try to redirect the resolution of dist/index.js back to the source file src/index.ts. This would be based on the rootDir, outDir and perhaps other options.

It does seem rather complicated though. esbuild would have to check every file to see whether it is a part of a referenced project within the context of the current tsconfig. The redirected source files would take precedence over built files, since those could be out of date.

I personally explored writing a plugin to resolve such cases, but using one of the workarounds seems like a better solution.

[electroma]

@nihalgonsalves thank you for sharing your experience!

We tried both workarounds too:

• "Paths" workaround does not seem to utilize TSC incremental build, and it may be slow at larger scale (in our case we live in a monorepo with about 100 modules)
• "main: index.ts" workaround looks like a hack, since other tools may not support it because Node.js is not TS-native environment and it also does not use TSC incremental build

For now the workaround we rely on is to use tsc -b as a preparation step and point ESBuild to the output index.js, that way we can have incremental cross-module compilation and still enjoy the speedy bundling from ESBuild.

Regarding support of project refs in ESBuild - I see your point, and I think that ESBuild probably can't and shouldn't play catch with mainstream TSC.

[evanw]

Is this something that other bundlers support? Such as Webpack, Parcel, Rollup, or Browserify? If other bundlers don't do this, then I don't see why esbuild should need to do this.

[electroma]

@evanw, it's already a relatively long thread, and in the original feature request I stated "Both rollup and webpack plugins have support for this feature". I have several modules using rollup and webpack and they both support incremental cross-module build out-of-the-box.

[evanw]

Both rollup and webpack plugins have support for this feature

I tried but I was unable to get the test case in #1250 (comment) to work in either Webpack or Rollup (or Parcel): https://github.com/evanw/esbuild-issue-1250. What am I missing?

[ggoodman]

@nihalgonsalves do you also have "paths" configured in tsconfig.json to allow mapping the referenced projects to source instead of to distributable files?

[jaredpalmer]

This is not a bundler problem. FWIW Thetsc -b command is a strange, arguable out-of-scope foray into build system land. Anyways, the way to solve this outside of tsc -b is with a dependency-aware task runner that knows how to execute tasks in topological order. yarn/npm/pnpm workspaces and/or lerna can do the latter in https://github.com/evanw/esbuild-issue-1250 if both packages in have specified a build script in their package.json. Then running lerna run build will ensure that pkg-b's build command is run before pkg-a's.

[ggoodman]

If pkg-a were to define the following in its tsconfig.json, I think it might 'work' without the need for external orchestration. I don't know if it might have an impact on the effectiveness of TypeScript's ability to decompose work along package boundaries. This is a pattern that I have been using with some success.

• tsconfig.json

   {
     "include": "./src",
     "paths": {
       // Tell the resolver to look for the source file instead of the distributable artifact
       "pkg-b": ["../pkg-b/src/index.ts"]
     },
     // referenced here too
     "references": [{ "path": "../pkg-b/tsconfig.json" }]
   }

One benefit from this approach that I like is that it avoids the duplication of helper functions that might otherwise find themselves injected in each intermediate build.

[nihalgonsalves]

@evanw turns out me writing that entire config in a comment wasn't perfect. I modified the example a bit with workspace/tsconfig fixes: 08ef5ba (#1). This is sufficient for tsc --build to work, as well as Webpack with ts-loader's projectReferences turned on. They both essentially use the same mechanism under the hood. (Needs yarn or npm 7 for workspace linking)

And as @ggoodman pointed out, what I mentioned in my initial comment was using paths as a workaround, and this works automatically since esbuild supports paths: bc3c877 (#1). You would usually put this into a base tsconfig and define alternative source paths (to replace the dist paths referred to in main or module) for each one of your workspaces.

Looking up the source file for a given dist/index.js (for example) would likely be possible by analysing a few compiler options (rootDir/include/files, outDir, etc), but I'm not sure if that's in scope for esbuild or just something a plugin should do.

Also: Rollup still doesn't work with this config, the official plugin seems to support neither project references, nor paths by default.

(When testing, make sure to delete the pkg-b/dist/ directory and .tsbuildinfo files, they would make esbuild work without paths, for example).

---

"Paths" workaround does not seem to utilize TSC incremental build, and it may be slow at larger scale (in our case we live in a monorepo with about 100 modules)

@electroma That's not true, if you combine it with project references it should still work for type-checking. And for esbuild this does not matter - it still gets through ~100k LoC in < 2 seconds.

---

@jaredpalmer You could also just run tsc --build and everything would work without workarounds – but then you lose esbuild's speed. Perhaps an idea would be to build using esbuild separately for each workspace using Lerna or wsrun --stages - though I feel that's even more complex than a simple paths workaround.

[jaredpalmer]

@nihalgonsalves tsc -b == tsc --build

[evanw]

Looking up the source file for a given dist/index.js (for example) would likely be possible by analysing a few compiler options (rootDir/include/files, outDir, etc), but I'm not sure if that's in scope for esbuild or just something a plugin should do.

This seems like a good thing to experiment with in a plugin to me.

[evanw]

This is not a bundler problem.

I'm going to close this then. It sounds like this needs to be solved in a layer on top of esbuild instead.

[smacker]

I have built a plugin for that: https://github.com/smacker/esbuild-plugin-ts-references
It works for my monorepo but I will be glad to know about other cases and make it more robust.

[skrat]

@smacker that's a good effort, unfortunately it can't be used with webpack's https://github.com/privatenumber/esbuild-loader

[pokey]

I was able to get this working using a custom exports field on all my monorepo package.json files, and then passing --conditions to esbuild:

  "exports": {
    ".": {
      "myOrganizationName:bundler": "./src/index.ts",
      "default": "./out/index.js"
    }
  }

Then

esbuild --conditions=myOrganizationName:bundler ...

[pokey]

Btw note that this issue doesn't just force you to make sure your local dependency packages are compiled. Pointing esbuild at the generated Javascript for your local dependency packages results in significantly higher bundle size than pointing esbuild at their raw Typescript sources:

➜ esbuild ./src/extension.ts --conditions=myOrganizationName:bundler --bundle --outfile=dist/extension.js --external:vscode --format=cjs --platform=node --minify

  dist/extension.js  636.0kb

⚡ Done in 82ms

➜ esbuild ./src/extension.ts --bundle --outfile=dist/extension.js --external:vscode --format=cjs --platform=node --minify 

  dist/extension.js  785.7kb

⚡ Done in 77ms