rootDirs should merge outputs

[justinfagnani]

Suggestion

🔍 Search Terms

rootDirs merge output

Related: #9875

✅ Viability Checklist

My suggestion meets these guidelines:

• This wouldn't be a breaking change in existing TypeScript/JavaScript code
• This wouldn't change the runtime behavior of existing JavaScript code
• This could be implemented without emitting different JS based on the types of the expressions
• This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, new syntax sugar for JS, etc.)
• This feature would agree with the rest of TypeScript's Design Goals.

⭐ Suggestion

rootDirs should merge outputs instead of emitting to the folders listed in rootDirs. This would be consistent with rootDir.

📃 Motivating Example

Starting point: single root

Let's say I start with a project with a src/ folder:

.
└── src
    ├── index.ts
    ├── lib
    │   └── lib.ts
    └── test
        └── test.ts

It has one rootDir and this config:

{
  "compilerOptions": {
    "outDir": "./",
    "rootDir": "./src",
  },
  "include": ["src/**/*.ts"],
  "exclude": []
}

When I compile, the output goes to ./ and mirrors the structure of ./src:

.
├── lib
│   └── lib.js
├── src
│   ├── index.ts
│   ├── lib
│   │   └── lib.ts
│   └── test
│       └── test.ts
└── test
    └── test.js

All well and good.

Friction point: add generated files

Now I want to add some generated .ts files to my project and keep them outside of src/ for easier source control.

I put a .graphql source file in my src/ tree, and set up the generator to output to gen/:

.
├── gen
│   └── lib
│       └── schema.ts
└── src
    ├── index.ts
    ├── lib
    │   ├── lib.ts
    │   └── schema.graphql
    └── test
        └── test.ts

Now I want the same output structure as before, but I want to compile the sources in gen/ and output them as if I had "rootDir": "./gen".

The first thing I try is to replace "rootDir": "./src" with "rootDirs": ["./src", "./gen"]:

{
  "compilerOptions": {
    "outDir": "./",
    "rootDirs": ["./src", "./gen"],
  },
  "include": ["src/**/*.ts", "gen/**/*.ts"],
  "exclude": []
}

but this unfortunately defaults rootDir to ./, putting src/ and /gen in the output paths and writing .js files as siblings to their .ts sources:

.
├── gen
│   └── lib
│       ├── schema.js
│       └── schema.ts
└── src
    ├── index.js
    ├── index.ts
    ├── lib
    │   ├── lib.js
    │   ├── lib.ts
    │   └── schema.graphql
    └── test
        ├── test.ts
        └── test.ts

This is very much not what I want or expect, and a big difference from the behavior of rootDir.

What I expect is that rootDirs acts like a multi-value rootDir, and for each input the root dir that it's in is stripped from the output path.

One additionally confusing aspect of rootDir vs rootDirs is that "rootDir": "./src" is not equivalent to "rootDirs": ["./src"]. The former will cause output files to be siblings to inputs.

It looks like to work around this we need to set "outDir": "./build", and then add an additional build step just to copy the files from ./build/src and ./build/gen into ./. This obviously can work but it's more of a complication than just a simple step: --watch will no longer work correctly unless you wire up the copy step to automatically run when ./build files change. For a project with simple build scripts, this is a pretty big leap in complexity.

Starting package.json:

  "scripts": {
    "build": "tsc --build"
  },

Desired addition of a generator:

  "scripts": {
    "build": "npm run build:graphql && npm run build:ts",
    "build:ts": "tsc --build",
    "build:graphql": "graphql-codegen"
  },

Actual addition of a generator, with broken --watch:

  "scripts": {
    "build": "npm run build:graphql && npm run build:ts && npm run build:copy",
    "build:ts": "tsc --build",
    "build:graphql": "graphql-codegen",
    "build:copy": "cp -r build/{src,gen}/*"
  },

Actual addition of a generator, with working --watch:

🤷‍♂️ I have to go figure this out still. Also, I'm not sure if this will badly interact with a monorepo, composite projects, and --build. The cross-package import paths are going to be different from the compiler output.

Potential workaround

I tried setting "rootDir": "./(src|gen)", hoping to tell tsc to strip those paths from the output. No luck as that path isn't interpreted as a pattern. That could potentially be a non-breaking way to add merging.

💻 Use Cases

Compiling generated files into a common output tree.

[justinfagnani]

Also, I'm not sure if this will badly interact with a monorepo, composite projects, and --build. The cross-package import paths are going to be different from the compiler output.

Well, so far it does look like adding the copy steps interferes with composite projects.

To test I made a change to a shared project that causes a break in a project that references it while I was running tsc --build --watch in the dependent project.

When I make the change to the source file, tsc recompiles, but with no errors. (Because the files actually imported by the dependent project are the copies and they haven't changed yet?) If I run the copy step the imported files do change, but the .tsbuildinfo file hasn't, so tsc doesn't see a change still. If I then kill the watch build and run tsc --build -f the error is picked up.

edit: More complications with a monorepo / composite setup. On a clean build, when a dependent project builds it'll run tsc on the referenced project, but not invoke the copy step, so the imports in the dependent project fail to resolve.

It seems like rootDirs and composite projects just don't mix? It'd be great to see a working example as requested in #37257 I think things would just work if rootDirs merged the roots as requested here.

[RyanCavanaugh]

Stating the obvious, probably: We can't just change this and break everyone's existing builds, so there'd have to be some new opt-in

Less obvious, perhaps: This is tricky. We'd start to have to deal with what to do with conflicting output names. It's also unclear how relative-path module specifiers should work in this world -- does importing "./foo" mean we'd scan every directory at the same relative point in the hierarchy, or would you only be able to import relative filenames from the input-side path despite there being a larger set of paths that would actually work at runtime?

[justinfagnani]

Yeah, changing rootDirs would be a breaking change, I was assuming there'd need to be an opt-in.

I think there are good answers to the questions you raise:

• Conflicting output names should be disallowed
• Relative imports should resolve within the merged folder hierarchy

This is how other src/gen overlay systems I've used work. It makes imports behave as if the generated files were generated directly into the source folders, which has the essential benefit of abstracting away whether a file is checked-in or generated.