Rethinking project layout

[remcohaszing]

Since we first started using types in JSDoc, TypeScript has evolved, Node.js has evolved, and also our team’s experience using TypeScript has evolved, including my own.

Below I have compiled a list of some general rules / learnings when working with TypeScript.

1. Each .d.ts file represents a .js file next to it in the same folder.
2. You’re supposed to use rootDir and outDir.
3. When working with JSDoc, rule 2 trumps rule 1.
4. TypeScript configuration files may have any arbritrary name.
5. The TypeScript language server looks up a special file tsconfig.json to provide editor features. This filename is also used by tsc by default if not specified as a CLI option.
6. Types in JSDoc can do a lot that can be done in .ts files.
7. Types in JSDoc can’t do everything that can be done in .ts files .
8. Humans are not supposed to author .d.ts files. Those should be generated by tsc.
9. Sometimes TypeScript produces unwanted output, and humans have to author .d.ts files.
10. TypeScript declaration maps are great, especially when working with types in JSDoc.
11. TypeScript files are supposed to be compiled. The compiled output is supposed to be run. You’re not supposed to run .ts files directly in Node.js. This one is not relevant for the unified ecosystem, but it is for me. :)
12. TypeScript project references are great, if you know how to use them.

The TypeScript 5.5 iteration plan contains 2 big new features that will make working with JSDoc a lot less limiting:

• @import tags for type imports.
• @nonnull tags as an alternative to TypeScript non-null assertions.

This leaves the following big limitation that unifiedjs projects still depend on:

• Typedefs are always exported.
• There is no way to re-export types from another file.

Based on all of the above, I suggest to make some changes to the setup of unifiedjs projects. Especially violating rule 2 causes some issues for which we have to use workarounds, such as violating rule 8.

Project layout

First of all, I suggest the following project layout:

package-name
├── lib
│   ├── exports.ts (optional)
│   ├── index.js
│   ├── tsconfig.json
│   └── package-name.js
├── package.json
├── script (optional)
│   └── some-script.js
├── complex-types.d.ts (optional)
├── test
│   └── some-test.js
└── tsconfig.json

lib

lib contains the source code.

lib/index.js

lib/index.js replaces index.js as the main export of the project. It re-exports all public members. Additionally, it contains all public type definitions, as TypeScript has no syntax for re-exporting types.

lib/package-name.js

This file contains the implementation JavaScript. This file could have any name really. Some places, such as the Chrome console, show the base name only. This is why I like to use a distinguishable base name. If there are no internal type definitions, we could also move everything into lib/index.js.

lib/exports.ts

JSDoc has limitations. Some things can only be defined in TypeScript, such as interfaces that may be augmented. If this is the case, we can add a file exports.ts. This file re-exports everything from lib/index.js, plus these additional types.

lib/tsconfig.json

This project defines how to compile the source code into type definitions.

{
  "compilerOptions": {
    "checkJs": true,
    // Because this project will be referenced (see below)
    // This also sets the rootDir to the "lib" directory
    "composite": true,
    // We already had this. I’m not really for or against it. It diverges from what most users do.
    "customConditions": ["development"],
    // We want to emit type declarations.
    "declaration": true,
    // We want to emit type declaration maps for an enhanced editor experience.
    "declarationMap": true,
    // We don’t want to emit JavaScript
    "emitDeclarationOnly": true,
    "exactOptionalPropertyTypes": true,
    // Exclude dom and worker types
    "lib": ["es2022"],
    "module": "nodenext",
    // We want to emit the type definitions to the "types" directory.
    "outDir": "../types",
    "strict": true
  }
}

Note that this sets module to nodenext. node16 and nodenext imply a different target.

Additionally we could set "types": [] to avoid loading all types from the node_modules/@types directory. Most projects don’t need them.

complex-types.d.ts

Although it is rare, sometimes TypeScript performs undesirable transforms. In this case we need to author .d.ts hands by hand. Since .d.ts files are not considered source files, they can exist outside the lib directory and still be referenced.

package.json

Since we break rule (1) in order to adhere to rule (2), we need point the types package export to the correct location. If we want to support the node10 module resolution additionally, we also need the top-level types field.

{
  "types": "./types/index.d.ts",
  "exports": {
    "types": "./types/index.d.ts",
    "default": "./lib/index.js"
  }
}

If the package needs exports.ts to define special types, package.json needs to reference that file instead.

{
  "types": "./types/exports.d.ts",
  "exports": {
    "types": "./types/exports.d.ts",
    "default": "./lib/index.js"
  }
}

We can omit the tsc --build --clean from the build script. This only existed as a workaround for breaking rule (2).

tsconfig.json

The package’s root tsconfig.json applies to everything that’s not part of the source code. This includes tests, scripts, examples, etc.

{
  "compilerOptions": {
    "checkJs": true,
    "customConditions": ["development"],
    "exactOptionalPropertyTypes": true,
    "lib": ["es2022"],
    "module": "nodenext",
    // These types do not need to be emitted.
    "noEmit": true,
    "strict": true
  },
  "exclude": [
    // We don’t want to type check files in the "coverage" directory.
    "coverage/",
    // The "lib" directory is type checked by "lib/tsconfig.json".
    "lib/",
    // This is needed for type-coverage.
    "node_modules/"
  ],
  // This TypeScript configuration depends on the build output of "lib/tsconfig.json"
  "references": [{"path": "./lib/tsconfig.json"}]
}

Monorepos

Project references allow setting up monorepos properly. For example, let’s use the MDX repo.

mdx
├── packages
│   ├── mdx
│   │   ├── lib
│   │   │   └── tsconfig.json
│   │   └── tsconfig.json
│   ├── react
│   │   ├── lib
│   │   │   └── tsconfig.json
│   │   └── tsconfig.json
│   └── rollup
│       ├── lib
│       │   └── tsconfig.json
│       └── tsconfig.json
└── tsconfig.json

packages/mdx/lib/tsconfig.json does not depend on any projects. It does not contain references.

packages/mdx/tsconfig.json depends on the types of mdx and react. it has the following references:

{
  "references": [
    {"path": "../react/lib/tsconfig.json"},
    {"path": "./lib/tsconfig.json"},
  ]
}

packages/react/lib/tsconfig.json does not depend on any projects. It does not contain references.

packages/react/tsconfig.json depends on the types of mdx and react. It has the following references:

{
  "references": [
    {"path": "../mdx/lib/tsconfig.json"},
    {"path": "./lib/tsconfig.json"},
  ]
}

packages/rollup/lib/tsconfig.json depends on the types of mdx. It has the following references:

{
  "references": [
    {"path": "../mdx/lib/tsconfig.json"}
  ]
}

packages/rollup/tsconfig.json depends on the types of mdx and rollup. It has the following references:

{
  "references": [
    {"path": "../mdx/lib/tsconfig.json"},
    {"path": "./lib/tsconfig.json"},
  ]
}

The top-level tsconfig is known as a solution file. It contains no file, but only references. It has the following content:

{
  "files": [],
  "references": [
    {"path": "./mdx/packages/tsconfig.json"},
    {"path": "./react/packages/tsconfig.json"},
    {"path": "./rollup/packages/tsconfig.json"}
  ]
}

Now running all of MDX would be a matter of running the following command from the project root:

tsc --build

But each package can also be built independantly. Also none if the packages conflict with each other because of globals anymore.

Type imports

Starting with TypeScript 5.5 we can use type imports. Current we use the following syntax:

/**
 * @typedef {import('./some-file.js').Options} Options
 */

This is not a type import, but a new equivalent exported type. It leads to loss of context, such as generics and documentation. it can’t be detected as unused. It also means we need to load some-file.d.ts, even if we don’t really use it, leading to more memory usage for TypeScript for dependants.

It’s better to start using the new TypeScript syntax:

/**
 * @import {Options} from './some-file.js'
 */

Non null assertions

Non-null assertions are safer than manual type casting. It happens relatively often that we know something is not nullish. In such a case casting string | number | null | undefined to string is unsafe, because it’s easy to forget one of the union members. Also other types of casting often involve needing to add additional type imports

One typedef per comment

I ran into some issues using @template tags in JSDoc comments that have multiple @typedef / @callback tags. Using one @typedef / @callback per comment solves this. I think it’s good to always use one type definition per comment for consistency.

Also sometimes I overlook where one type definition ends and another starts. The visual boundary of closing and starting a new comment helps with this.

---

I hope the above clarifies some of the problems in the current TypeScript setup and the ideas how to solve them. None of these things need to happen overnight. Most changes don’t even affect users.

Also not everything is set in stone. Some variations include:

• Use different file names.
• Don’t use index.js. This was a special file name in Node.js CJS. In ESM it’s not.
• Define all types in .ts files. Types in JSDoc do have some limitations.
• Re-use a shareable config, as I proposed in https://github.com/orgs/unifiedjs/discussions/199, but later questioned myself.

One effect of these changes I didn’t think of is that this tests the user facing type definitions. Trying to make this work in the vfile repo I accidentally discovered a bug in the TypeScript emit.

[wooorm]

8. Humans are not supposed to author .d.ts files. Those should be generated by tsc.

This is the main thing I strongly disagree with.

TS (the syntax) and JSDoc (the TS syntax) consist of 2 things:

a) annotations; in some future we can move to types-as-comments
b) definitions; (and other complex things that have little to do with JS values)

As you also correctly note: TS tries to generate smart JS code and smart .d.ts types; which is nice to start with but it fails, particularly when doing smart and complex things and you want to make things fast for billions of users.
I often find that TS is generating broken types. With import('.../node_modules/..') stuff for example.
A similar thing happened with CoffeeScript: nice syntax sugar for loops but then the code is a bit slow and you’d much rather use, well, a while loop.

Adjacently, I’d like to start using interfaces. And extending those. And JSDoc doesn’t have them (and xo doesn’t like them). But because of stuff like this: https://www.totaltypescript.com/react-apps-ts-performance. And also my own experience currently with generating docs from types: I’d like to move definitions up instead of getting them from deeper files.

We could improve a lot of our performance by hand writing our type definitions.
Then the JS APIs can use type annotations with JSDocs.

So, something like:

package-name
├── lib
│   ├── ...whatever
│   └── tsconfig.json // If you want this, sure, buy why?
├── index.js // Export the JS API from `lib/`
├── index.d.ts // Export the JS API from `lib/` *and* define the `interface`s and complex type things
├── package.json
└── tsconfig.json

Project references allow setting up monorepos properly. For example, let’s use the MDX repo.

MDX is already working without project references? So what do project references improve?

Type imports

Yeah sure.

Non null assertions

Why not devlop.

One typedef per comment

Yep. Even without @template. But see also my main point.

[remcohaszing]

I think you’re missing the main point: rules 1-3. You’re supposed to use rootDir and outDir. I’ve seen several mentions of this by the TypeScript team, but here’s one I recall: microsoft/TypeScript#53315 (comment)

We’re seeing a lot of symptoms from not leveraging this:

• Having to run tsc --build --clean before tsc --build, slowing down compilation
• Persistent errors in the editor
• Emitting redunant .d.ts files
• Unnecessarily complex include / exclude patterns in tsconfig.json
• The need to author .d.ts files

Humans are not supposed to author .d.ts files. Those should be generated by tsc.
This is the main thing I strongly disagree with.

It’s true though. This is for example why .d.ts files are not copied from into outDir, and why skipLibCheck means to ignore type checking .d.ts files. .d.ts files are however supposed to be human readable. Because you’re not supposed to, doesn’t mean you’re not allowed to. But it’s more risky and use cases are rare.

I often find that TS is generating broken types. With import('.../node_modules/..') stuff for example.

I think I’ve seen this only once, when you linked it to me. I wouldn’t be surprised if this is another symptom of not using rootDir / outDir somehow.

Adjacently, I’d like to start using interfaces. And extending those. And JSDoc doesn’t have them (and xo doesn’t like them). But because of stuff like this: https://www.totaltypescript.com/react-apps-ts-performance.

I personally prefer interfaces, but I don’t think this difference is that impactful. Much more impactful is the choide to redefine and export all imported types locally. Using @import will solve a lot of that.

And also my own experience currently with generating docs from types: I’d like to move definitions up instead of getting them from deeper files.

This aligns with one of the variations I mentioned:

• Define all types in .ts files.

Project references allow setting up monorepos properly. For example, let’s use the MDX repo.

MDX is already working without project references? So what do project references improve?

It kind of works by accident now.

• It uses a single tsconfig.json file
• It contains an unused tsconfig.json file in every project directory
• It doesn’t leverage rootDir and outDir yet
• It uses the dom types for all projects, even those that don’t run in the browser.
• It uses the node types for all projects, even those that run in the browser.

We’re just lucky there aren’t any conflicts currently.

Non null assertions

Why not devlop.

Depends on the situation. Both have their place.

[wooorm]

I think you’re missing the main point: rules 1-3

I’m disagreeing with you on points.
You came up with your 12 rules.
I’m noticing things maintaining hundreds of projects too:
TS is bad at generating types from JS and JSDoc isn’t the place to define TS-only things.

Your ideas are very big, and it’s only a discussion instead of a concrete PR, so I indeed do not know what you think is important and what isn’t.

It’s true though. […]

I don’t think that the rest of this paragraph supports your conclusion.
We don’t use outDir or skipLibCheck.
It also conflicts with what I’m seeing, from my previous point: TS is bad at generating types from JS, and JSDoc isn’t the place to define TS-only things.

I think I’ve seen this only once

I notice it often. Currently, again, all remark lint rules that accept options broken for example: https://www.npmjs.com/package/remark-lint-checkbox-character-style?activeTab=code

If you have a theory that it might somehow be fixed, then that theory should be tested in practice.

I don’t think this difference is that impactful.

The post describes a measured big performance improvement. You can’t just say “no it’s not faster” without any evidence?
If what you think is true, that it is not much faster, then I argue that the ecosystem is so big, and has so many users, that improving our types a little, will still have a big impact.

Much more impactful is the choide to redefine and export all imported types locally. Using @import will solve a lot of that.

Writing the types in .d.ts will also do that.

This aligns with one of the variations I mentioned:

Define all types in .ts files.

Indeed, we draw the same conclusion, and I diverge that we can use .d.ts files and not use .ts files because we have .js files, and in those .ts/.d.ts files we never want to have “actual values”, only types.

It kind of works by accident now.

I don’t understand how can you describe it as accidental, when it is intentional to work.

It contains an unused tsconfig.json

I assume that some tool instead uses it?

It uses the dom types for all projects, even those that don’t run in the browser.

Where?
Because there’s a <reference in docs?

We’re just lucky there aren’t any conflicts currently.

What conflicts could there be?
Why don’t micromark, remark-lint, remark/rehype/retext, rehype-minify, and this, don’t have those conflicts? Or do they?

Both have their place.

I don’t really see it. Anyway, sure, when TS x.y adds a new feature, then sure we can use that.