Add ability to configure TypeScript Node from tsconfig.json

[blakeembrey]

Since the module could end up being used in a lot of different environments (E.g. gulp), there should be a way to configure it via config file or environment. Since tsconfig.json already exists, I'd say we piggy back off it and add a typescript-node property to the object.

[ghost23]

Hi,

in the source code I see, that the tsconfig.json is already being read. This leads to some issues with my project. My project looks like this:

/src/my/typescript/**/files.ts
/test/tests.ts
/tsconfig.json

When running tsc here everythings works as expected. But when I run mocha --compilers ts:ts-node/register it also loads the tsconfig.json and gets confused with the paths, because the tsconfig says, all ts files would be located in src. But obviously the tests aren't. Also it builds the paths wrong, e.g. it searches in: /tsconfig.json/src/../.. So the tsconfig.json filename is in the path.

If I temporarily rename the tsconfig.json to something different and run mocha again, everything works just fine.

So I guess, having the ability to ignore the tsconfig for ts-node would be nice, although I am not sure, how this could be done in conjunction with mocha. Do you have any suggestion?

[blakeembrey]

@ghost23 Can you please elaborate on the issue you're having. It's likely you're having a different issue, but I don't have enough details to understand what it is. I'm using it with tests all the time, and I absolutely need to load from tsconfig.json to make sure I load definition files. I'll add a --no-project option, but it's not really the correct fix here.

You say that tsconfig.json is in the paths. How did you find that? All the tsconfig.json parsing logic is https://github.com/TypeStrong/tsconfig/blob/master/src/tsconfig.ts#L195-L213 and I haven't seen that before.

[ghost23]

Hi, when I run the mocha command, I get the following error, that's why I said, it is in the path:

mocha --compilers ts:ts-node/register

⨯ Unable to compile TypeScript

File 'C:/Users/me/projects/ts-test/src/analysis/body_traverser.ts' is not under 'rootDir' 'C:/Users/me/projects/ts-test/tsconfig.json/src'. 'rootDir' is expected to contain all source files. (6059)
File 'C:/Users/me/projects/ts-test/src/analysis/node_worker.ts' is not under 'rootDir' 'C:/Users/me/projects/ts-test/tsconfig.json/src'. 'rootDir' is expected to contain all source files. (6059)
File 'C:/Users/me/projects/ts-test/src/analysis/scope_detector.ts' is not under 'rootDir' 'C:/Users/me/projects/ts-test/tsconfig.json/src'. 'rootDir' is expected to contain all source files. (6059)
File 'C:/Users/me/projects/ts-test/test/test.ts' is not under 'rootDir' 'C:/Users/me/projects/ts-test/tsconfig.json/src'. 'rootDir' is expected to contain all source files. (6059)

npm ERR! Test failed. See above for more details.

You see, the first three files really are in the src directory, whereas the last one is the test.ts itself, which is not in the src directory, but the test directory. Anyway, the errors state, that the path for rootDir is expected to be ..../projects/ts-test/tsconfig.json/src

My tsconfig.json file looks like this:

{
"compilerOptions": {
"module": "commonjs",
"noImplicitAny": true,
"listFiles": true,
"removeComments": true,
"preserveConstEnums": true,
"rootDir": "./src",
"outDir": "./built",
"sourceMap": true
}
}

[blakeembrey]

@ghost23 Thanks for the response, looks like tsconfig might not be handling rootDir properly when there's no files specified. However, this is a TypeScript error caused by the files outside of rootDir. The only fix here is to add --no-project, which I'll do now, or to change rootDir option.

It also might be worth it, from ts-node, to override that option because it's invalid here - thoughts on that?

[blakeembrey]

As far as I can see, tsconfig is correct in loading everything still. The tsconfig file is a bit of a mess, to be honest, and doesn't work how I'd expect it to. If I recall, I did the same thing writing typings and found the easiest change was to move tsconfig.json into src/ otherwise tsc finds everything and gives the same error. That may be a bug in tsc though. Have you tried compiling with tsc?

[ghost23]

Yes, when I compile with tsc, everything works as expected. Of course, I am not compiling the tests then, which is the way I want it. I only want the tests to compile on the fly during the mocha test.

[blakeembrey]

@ghost23 Weird, and you don't have a files array at all? I'd say that is definitely have a bug in tsconfig then, but I'm going to quickly replicate it. Then I'll patch ts-node with a couple of new options, but I think the best approach for ts-node is to override the rootDir options - it's only used for compilation as far as I can tell.

[ghost23]

The way I understand it is, when you run tsc only with rootDir, it takes everything and ONLY everything in that directory and compiles it. So no need for a files array. Quite comfortable in my view :)

[blakeembrey]

@ghost23 Right, I understand. It's just last time I tried that, it wasn't working and I'm trying to recall why. Most likely, it was the usage of the typings/ directory in the root which was breaking things and making rootDir unusable. It's based on input files, and not output structure, which was the issue. Anyway, it's unrelated, was just trying to find the issues that exist in my head.

[ghost23]

Oh, I forgot to mention that I also have a typings directory in root. Sorry. Now that you say that, I begin to understand your point. It is indeed weird, that tsc is OK with looking for typings outside of rootDir. But it works.

[ghost23]

Just in case the question comes up, I am using tsc 1.6.2

[blakeembrey]

Awesome, thanks. It seems if you're using definition files only outside of root, it's ok. Not sure if that's new, or I had some other issue, but it makes sense. How is tsc not crashing on the test file of yours though? If I put any .ts files outside of rootDir, I get the same error.

➜  tsc-rootdir-test  tsc               
error TS6059: File 'test/test.ts' is not under 'rootDir' 'src'. 'rootDir' is expected to contain all source files.
➜  tsc-rootdir-test  ls
dist          foo.d.ts      src           test          tsconfig.json

[ghost23]

As I said, I am not compiling the test files with tsc. tsc is ignoring the test folder, because it is outside rootDir. I only want to compile the test files during the mocha run.

And when I simply remove or rename tsconfig.json and run mocha --compilers ts:ts-node/register in the root folder, it just magically works. But not, if the tsconfig.json file is present.

[blakeembrey]

@ghost23 I understand, but how are they ignored? Mine aren't ignored by default. Are you using exclude? I'm trying to track down the core behaviour of tsc and tsconfig here so that tsconfig can be updated if there's a bug.

Edit: Basically, I'm replicating what you have told me above and running tsc, but seeing a different result.

Edit 2:

➜  tsc-rootdir-test  tsc
error TS6059: File 'test/test.ts' is not under 'rootDir' 'src'. 'rootDir' is expected to contain all source files.
/Users/blakeembrey/.npm-packages/lib/node_modules/typescript/lib/lib.d.ts
foo.d.ts
src/bar.ts
test/test.ts
➜  tsc-rootdir-test  tsc --version
message TS6029: Version 1.6.2

[ghost23]

Gee, I am ... something. I actually also get the error, but since I have some more files, the error quickly disappeard just behind the window. So yes, I do have the error, but tsc still compiles all the other files just fine. I guess, with an exclude the problem could be fixed.

[ghost23]

yes,

"exclude": [
        "test"
]

fixes the error

[blakeembrey]

Awesome, I just pushed v0.5.0 anyway. It has a noProject flag and sets rootDir to the current working directory by default.

[ghost23]

Great, thanks for your quick responses. How would I use that in conjunction with mocha?

[blakeembrey]

You'd have to use the executable - ts-node --noProject node_modules/.bin/mocha. There's no other way to use options yet.

[ghost23]

ah, ok. I see. will try that. Thank you.

[blakeembrey]

@ghost23 It should have "just worked" for you too, since it overrides the rootDir option.

[ghost23]

Ok, I tried it with ts-node --noProject node_modules/mocha/bin/mocha, but it didn't work. I got weird error messages, which indicate, that typescript had not been transpiled first and so it gave errors for "import ..." statements and the like. Perhaps mocha is programmatically running normal node unless i pass the compilers statement?!

[blakeembrey]

@ghost23 Replace mocha with _mocha. Sorry.

[willm]

I don't know if this still belongs in this issue, but i have the following tscconfig:

{
    "compilerOptions": {
        "module": "commonjs",
        "target": "es5",
        "noImplicitAny": false,
        "sourceMap": false,
        "outDir": "dist"
    },
    "files": [
        "basket/get.ts"
    ],
    "exclude": [
        "node_modules",
        "dist"
    ]
}

when I run tsc, my source files are in the correct place ./dist, however when i use ts-node with mocha in any of the ways suggested, the compiled javascript files are saved in the same directory as the typescript files.

I've tried:

ts-node --noProject node_modules/.bin/_mocha
test": "mocha --require ts-node/register test/**/*-test.ts

[blakeembrey]

@willm Sounds like you're doing something else, ts-node does not generate any files.

[bertolo1988]

I am having /test/test.ts' is not under 'rootDir'...'rootDir' is expected to contain all source files..

The script i am running is `"test": "mocha -r ts-node/register test/test.ts",``

Any solution?

[blakeembrey]

Why don't you just make your tsconfig.json file the one that works and add a more specific tsconfig.es2015.json used elsewhere? There'll always be cases where you need a config for two different reasons, and this is probably a more effective solution long term than adding overrides to tsconfig.json because the next issue would be handling that with extends.

[cspotcode]

I'm working on a PR that loads config flags from tsconfig.json.

In the process, I'm generating a JSON schema from the TS types via typescript-json-schema. So far the experience is nice:

The JSON schema generator can extract arbitrary @-tags into the schema, which means it could hypothetically drive the CLI parser or render the README, too. Not sure it's worth the effort, but it's cool that it's possible.

[G-Rath]

@cspotcode just FYI schemastore has a schema for tsconfig.json 🙂

That schema tends to be the default used by IDEs & other tools these days, and schemastore will be happy to include the definition of the ts-node property in their schema.

[cspotcode]

@G-Rath very cool, that sounds great. Right now the schema is extending https://schemastore.azurewebsites.net/schemas/json/tsconfig.json via a oneOf declaration.

I found something suggesting that VSCode used that schema by default. Is that a Microsoft mirror of schemastore or something?

[G-Rath]

@cspotcode that is schemastore 😄

They host all their schemas, so you don't have to download them; instead you just point to the URL in $schema.

https://schemastore.azurewebsites.net/json/

(MS actually use that schemastore as the official schema, which is one of the reasons why it's such hig quality)

[cspotcode]

Ok, I wasn't sure if it was the same as http://json.schemastore.org/tsconfig.

We should still bundle the schema with ts-node in case there are changes, and users want to use the version of the schema built for a specific version of ts-node. But it'll be sweet having a default that "just works" for everyone.

[cspotcode]

A few notes about the implementation:

Config order of precedence

We're adding a third source of config flags: env vars, argv flags, and now tsconfig. They override each other in this order:
Env vars are overridden by anything from tsconfig.json, which is overridden by --flags.
We check env vars and flags for a --project flag, then we load or skip tsconfig accordingly. Finally, we merge all 3 sources of options.

---

Question: Should tsconfig be allowed to specify skipProject and/or project options?

It's potentially confusing, but I can see 2 cases where it might be useful:

a) Your tsconfig.json must be configured with the flags you want your code editor to use. There's no easy way to point the language service at a different config file. However, you want ts-node to use include, exclude, and files options that are different than what you have in tsconfig.json. So you specify a typescript-node.project flag that re-points ts-node to a different tsconfig-tsnode.json.

b) You want ts-node to ignore all the typescript options specified in your tsconfig.json. You add typescript-node.skipProject to your tsconfig to achieve this behavior.

---

Question: should "emit" docs be updated to say they'll emit to outDir, not necessarily .ts-node?

Seems like a small oversight in the docs. I didn't realize ts-node's emit option would respect my outDir config. This seems awesome, because I can pair it with files: true and transpileOnly: true to have a super-quick build step.

[cspotcode]

Quirk: Different compiler used to read tsconfig than to compile

readConfig() requires a loaded compiler and a cwd to search. Therefore, readConfig will use typescript by default. If tsconfig specifies an alternative such as ttypescript, then it will be used for compilation but not for config loading. I think this should be fine since most alternative compilers require typescript as a peer dependency anyway, and even if they don't, it shouldn't be trouble for the user to install it as a dependency.

This could impose a performance hit, since we're loading 2x compilers at startup. But again, if ttypescript is internally loading typescript anyway, it's not actually slower.

[blakeembrey]

Quirk: Different compiler used to read tsconfig than to compile

This isn't currently true, is something changing here? Currently it passes ts around.

Question: should "emit" docs be updated to say they'll emit to outDir, not necessarily .ts-node?

Possibly. To be fair, the emit option is only something on master I was toying with to try and improve performance. You should base your branch off 8.x for now.

Question: Should tsconfig be allowed to specify skipProject and/or project options?

Let's not support it today, it does seem confusing. At least we could always add it later.

[blakeembrey]

@cspotcode We might want to rename it to the ts-node property, typescript-node is what the project was called when I first wrote it and the issue was probably created around that time.

[cspotcode]

@blakeembrey

We might want to rename it to the ts-node property

Awesome, I like that better. I'll go with "ts-node".

Quirk: Different compiler used to read tsconfig than to compile

This isn't currently true, is something changing here? Currently it passes ts around.

Currently ts-node is able to read the compiler option and load the compiler before loading the tsconfig file. With this change, the tsconfig file will be setting option values. So we have a chicken-and-egg problem: if the tsconfig specifies an alternative compiler, we won't know about it until after we've loaded a compiler.

[blakeembrey]

Right, that makes sense. Maybe we can omit that functionality until it’s requested? It doesn’t sound ideal to load twice.

[G-Rath]

Random drive-by:

Env vars are overridden by anything from tsconfig.json, which is overridden by --flags

Is there a particular reason for this? I feel like it might be better for env variables to override tsconfig.json, b/c that way I can override single values from the tsconfig.json that ts-node magically finds w/o having to actually setup a whole new tsconfig.

This could be a moot point b/c technically the configuration for ts-node is set within tsconfig so it's a bit of a snake eating its own tail, but I feel like there could be some value in being able to "comfortably" override things via env without having to worry on if a setting is set in the tsconfig, or even having to track down the tsconfig that'll be loaded to check in the first place 🤷‍♂

[cspotcode]

@blakeembrey To make sure we're on the same page, I think we agree on the following implementation:

If --compiler or TS_NODE_COMPILER specify an alternate compiler, use it to load tsconfig and compile. Otherwise use standard typescript compiler to load tsconfig. (same as today)
If tsconfig.json specifies an alternate compiler, load and use it for everything except loading the config file, which we have already done at that point.

The above implementation assumes that the custom compiler does not change the behavior of config file discovery or parsing. I'm pretty sure this is true for every custom compiler I know about. (byots, ttypescript, ts-patch??)

Also, this implementation remains fast for the common case: using the default compiler.

If users of custom compilers complain about a slight startup cost, we can explain this chicken-and-egg problem and recommend that they use the --compiler flag or env var. They'll have a pretty straightforward workaround.

We could also add a flag loadTsConfigWithCustomCompiler which would force ts-node to re-parse tsconfig using the custom compiler, but I would only want to implement that if/when users have a compelling use-case.

---

One other quirk: users won't be able to specify ts-node options in extended tsconfig files. They'll have to put everything in their primary tsconfig file. I think this is totally fine.

[cspotcode]

@G-Rath I'm not opposed to that. My goal with this feature is to avoid using environment variables, so I admittedly don't think about the UX of env vars much. I'd rather avoid them. How do people typically use the env vars? Do they set them in their bash profile? Do they set them via package.json "scripts"? The former would be very problematic if it override tsconfig; the latter seems more in line with what you want.

Today, TS_NODE_COMPILER_OPTIONS or --compilerOptions will override whatever comes from tsconfig, so that's one example of env vars overriding tsconfig.

Can we find a situations where we'd want to override options via env vars, and we could not easily use --flags?

[blakeembrey]

If tsconfig.json specifies an alternate compiler, load and use it for everything except loading the config file, which we have already done at that point.

I’m wondering if we can just remove compiler overrides from the tsconfig to start with altogether. Do you have a use-case for it already? If not, let’s delay supporting it in the initial version to keep the implementation simpler.

Also agree that extended config options aren’t an issue for this version. I can’t see anyone making a request for it either.

[blakeembrey]

On the env var debate, I don’t have too much of an opinion. I think if the goal is to get this in without too many changes, the easiest place to apply these options before flags and environment variables.

[cspotcode]

@blakeembrey

Do you have a use-case for it already?

ttypescript comes to mind. The tsconfig would look like this:

{
    "ts-node": {
        "compiler": "ttypescript"
    },
    "compilerOptions": {
        "plugins": [
            { "transform": "ts-transformer-keys/transformer" },
        ]
    }
}

...and everything would "just work." The user can put #!/usr/bin/env ts-node on their scripts, and use ts-transformer-keys. As I understand it, the ttypescript compiler loads the transformers so ts-node's transformers option isn't involved.

[G-Rath]

Can we find a situations where we'd want to override options via env vars, and we could not easily use --flags?

When we're using a tool that uses ts-node but doesn't give us an option for specifying configuration, (which is fine, b/c ts-node has env variables + soon tsconfig).

In my mind being able to do TS_NODE_PROJECT="tsconfig.eslint.json" && eslint --config .eslintrc.ts or TS_NODE_PROJECT="tsconfig.webpack.json" && wp --config webpack.dev.ts gives a nice peace of mind that ts-node will do exactly as you've just explicitly told it to in a form as close to flags as you can get w/o actually being able to pass flags.

In saying all that....

How do people typically use the env vars? Do they set them in their bash profile? Do they set them via package.json "scripts"? The former would be very problematic if it override tsconfig; the latter seems more in line with what you want.

You make a very good point - I think it might be better to have tsconfig over env; but also could be worth just not specifying it explicitly (per say), in favor of "whatever's easier for now, and see who complains/asks for this feature"?

Personally, my two wishes are: the compiler example you gave, since I use ts-transformer-paths and so would be super useful to have ts-node automatically use ttsc, and to have the ability to specify the tsconfig that's used, like in the two examples above.

I've not followed this issue a lot, so you might have already answered this but where does TS_NODE_PROJECT fit into this issue?

[cspotcode]

@G-Rath Thanks, I think your use-case will be easy regardless of which way the env-var debate goes.

TS_NODE_PROJECT / --project will not be read from tsconfig.json, because that'd be weird: we'd load a tsconfig.json, it'd tell us to load a different config, and we'd totally ignore the first one. I initially proposed allowing this, but @blakeembrey said we shouldn't, and I agree.

For your example, TS_NODE_PROJECT=tsconfig.webpack.json wp --config webpack.dev.ts will work as you expect. I assume your tsconfig.webpack.json might look something like this:

{
  "ts-node": {
    "transpileOnly": true,
    "compiler": "ttypescript"
  },
  "compilerOptions": {
    "resolveJsonModules": true,
    "plugins": [{
      "name": "ts-transformer-paths"
    }]
  }
}

EDIT: forgot to include ttypescript and ts-transformer-paths in the example.