support globs in tsconfig.json files property (or just file/directory exclusions)

[JeroMiya]

An extension to #1692

Our ts project uses a grunt-based build system, with bower dependencies for the application and node_modules for build-tools. We will need the ability to exclude node_modules and bower_components from the build by default (unless files are referenced via /// ).

This causes an issue: we have end-to-end tests running in protractor, as well as unit tests running in jasmine, and unless we have the ability to exclude node_modules and bower_components, both will be parsed and we'll get duplicate definition errors.

In Visual Studio today, this is why we need to exclude both bower_components and node_modules from the project, because visual studio automatically parses all files in the project with no way to exclude them except to keep them out of the project altogether.

A number of third party tools have started to support tsconfig.json, and one thing I've noticed is that most of them support some sort of extension to tsconfig.json that supports file globs. I propose we just add support into the spec for file globs. For the version of typescript on npm, we can just depend on the appropriate file glob libraries in the npm ecosystem. For the Visual Studio embedded version of TypeScript, just embed the glob libraries that are needed statically in the installation.

As a fallback, if the notion of including a third party glob library is a non-starter, then the following fallback supporting a subset of the glob patterns would solve our problem at least partially.

Fallback:
Support both file and directory inclusions and exclusions in the files property:

files: [
  'app/', (equivalent to 'app/**/*.ts')
  '!e2e/', (equivalent to '!e2e/**/*.ts')
  '!node_modules/',
  '!app/bower_components/',
  '!app/vendor/fooLib/fooLib.ts'
]

[danquirk]

We certainly talked about this for awhile. See Anders' comment here for one large concern: #1692 (comment)

[JeroMiya]

Yes, I've read that comment. Unfortunately, for larger and more complicated projects, the lack of these glob and exclusion patterns makes it impossible to use tsconfig.json at all. If editors support tsconfig.json, we would end up having to disable it due to various errors in mismatched configuration between e2e, unit tests, and the application. And, if the editor uses tsconfig.json to configure compile-on-save and auto-complete in a standard way (and doesn't support a way to override this in a custom project file format, or extension to tsconfig.json as is the case with atom-typescript for example), then we've lost all of that functionality as well.

Globs and exclusion patterns are just how large typescript projects are made managable. We manage it now with build tools like grunt and gulp which support globs and exclusions, but so far the editors and IDEs haven't been configurable in the same way, leading to messy workarounds. For example, in visual studio we have to exclude all the end to end tests from the main app project so it doesn't parse them and complain about duplicate identifiers. We then have a separate visual studio project which just includes the end to end tests. We should be doing the same thing with unit tests, but we decided not having jasmine type declarations visible from the application code was not worth creating a separate unit test project. I would rather edit files in the project all at once, like I would do in Sublime or Atom or Brackets.

[NoelAbrahams]

@JeroMiya, perhaps the real problem here is how the projects are organised. I would consider moving the tests (both unit and end-to-end) to a separate project(s). IMO it's a better practice to keep tests and application code separate.

[JeroMiya]

There are many style guides out there advocating for grouping files by feature. We've tried it both ways and find that grouping things by feature instead of by file type helps you be more productive, especially for larger projects with hundreds of features. You tend to work on a particular feature at a time, and it's a real pain to track down all the files related to that feature when they're spread out across multiple root directories. This way also encourages developers to write unit and end to end tests at the same time as changes to the code. I highly recommend it over keeping test code separate from the application code.

Regardless, as a compiler, TypeScript should not be opinionated about project file organization.

[NoelAbrahams]

Fair enough. We tried the grouping files by feature before moving the test cases out. Having test code in the same place as application code made it difficult to prototype features, because a lot of time was spent fixing compilation errors in the test cases.

[basarat]

FWIW atom-typescript supports filesGlob as an optional extension to files. See spec.

[NoelAbrahams]

@basarat,

    "filesGlob": [
        "./**/*.ts",
        "!node_modules/**/*.ts"
    ]

That looks like a nice feature, because it actually plays well with "files". I wonder what the objection from the TS team is to including this.

Perhaps the argument is: One cannot post the tsconfig to someone else to say "Hey, here's a description of all the files in my project".

Edit (after morning tea)

quote from @ahejlsberg

(1) Let me drop an empty tsconfig.json file in a directory and have everything in there be a project, and (2) let me describe the exact list of files I want included. The current design covers both with a minimum of complexity. That said, I can certainly see how an "exclude" list could be useful, although I suspect we'd then need globbing as well, so the level of complexity goes up. I definitely don't want this growing into an alternate build system. The intent is more to offer tools (e.g. editors) a simple and deterministic way to decide what the compilation context for a particular file is.

[basarat]

Just a note: whenever atom-typescript reads a file with filesGlob it expands the files in 'tsconfig.json' file so that people that are not using atom-ts will still get the correct list of files.

[rcollette]

+1
Globs are so common in use now. There could be .ts files in bower components under my project which have already been compiled to js and may have been done with a different release of TS so I wouldn't want them recompiling. I guess exclusion would handle the case but it just seems like once you support globs you won't need anything else.

I think leaving this up to an IDE to edit, like atom-typescript, is a bad idea. You're basically asking every IDE to implement the logic to keep the tsconfig.json file up to date.

[bidiboom]

I'm in the same case as @JeroMiya , complexe project (e2e, unit test and project code). The outDir option is really good to this kind of project so we can cleanly separate our TS code from the produce code : no accidental JS code modification by the team for example. And this glob option is almost mandatory for our kind of project due the number of file. Thanks to Gulp, we can handle this point but it can be really good to have this glob supported by the tscong file

[panuhorsmalahti]

Globs are definitely needed. I don't see how I could manage the complex, multi-package TypeScript-based platform at work without globbing. Referencing every file by hand would be madness and require large amount of unnecessary work.

Globbing is especially useful when working with npm. I can depend on a npm package (which includes the type definitions), and then glob them in without knowing about the internals of that package.

Using the current setup in our project I can simply build package A, which consists of multiple modules, then require it from package B and glob the type definitions without any manual work even if the file structure of package A changes.

In theory the package A could output a single type definition file, but this is not supported by TypeScript, as each module of package A will create it's on .d.ts file. I already had to do a bit magic by writing a sort of "compiler" to make the .d.ts files ambient external declaration files automatically.

[park9140]

@ahejlsberg, @basarat, a few thoughts on this.

One major issue with, .net projects in general, is with source control systems and project files. By inserting a file reference into your project file every time a new file is added to a folder you end up with additional and unnecessary churn in your repository.

On larger projects with multiple teams working on them you constantly end up with conflicting changes to files like tsconfig. As such in addition to unnecessary file churn we now have additional labor to manually merge changes to a file that could be automatically handled.

[basarat]

As such in addition to unnecessary file churn we now have additional labor to manually merge changes to a file that could be automatically handled.

Agreed and would be great to have filesGlob supported.

FWIW If its a merge commit just delete the files section and let atom-typescript regenerate it for you. If its a rebase I chose "keep both" at each level ;). YMMV. Definitely needs glob support.

[mhegazy]

We would be open to taking a PR for excludes folders.

Our problem with globs is the complexity to support it, and we usually do not like to take dependencies on other packages as it limits our interoperability and system-independence. Having said so I think it is an important feature and indead does add value. Would be open to proposals here.

[basarat]

So no for stuff like : https://github.com/aspnet/Home/wiki/Project.json-file#sources ?

FWIW I don't have a big issue with the workaround we've managed :). An updating project file isn't a problem new or annoying to me personally.

[johnpapa]

tldr; Wow, we need this to support glob patterns.

Here is my use case ... I have a project with a few 100 files in TS and several 3rd party libraries with a concatenated lib.d.ts file. I want intellisense everywhere with the least amount of friction.

Option 1) Add /// reference to every file to point to the lib.d.ts and leave the tsconfig's file array undefined so it reads all of my app TS. Downside ... every file has this extra line ... yuk (100's of places)

Option 2) Add a gulp task that creates the files array in the tsconfig.json with the 3rd party lib.d.ts and my .ts files (lots of them). Downside, a gulp task to keep up, but at least my code is untouched

Option 3) Add glob support to tsconfig's files array so I can do this

files: ['lib.d.ts', '**/*.ts`]

I vote for option 3. So much easier and less friction for devs.

[dmorosinotto]

+1 for globs in files with syntax for exclusion
files: ["lib.d.ts", "**/*.ts", "!node_modules/**/*.ts"]

[elishnevsky]

I see this issue is still open. Is support for glob patterns in tsconfig.json still being considered? That would make so many developers happy.

[CyrusNajmabadi]

Is there a standard file-globbing specification anywhere? If we do this, we'll need to ensure that the globbing works the same on unix/windows, and that likely means writing the code to understand it ourselves.

[danquirk]

Ideally we could just use some existing thing like https://www.npmjs.com/package/glob rather than rolling our own :(

[CyrusNajmabadi]

Ideally. But can we use such packages from tsc.exe where we don't have access to node packages?

[ahejlsberg]

Right, the issue is that tsc.exe and the VS language service aren't in a position to take a dependency on node.js and therefore we can't use the node.js glob package. My feeling is that our best option so far is what is suggested in #3043: An exclude property in tsconfig.json that specifies a list of files and folders to exclude when a files list isn't explicitly specified. The net effect is the same as globbing with the list ["**/*.ts"] followed by a bunch of excludes. For example

{
    exclude: ["node_modules", "tests"]
}

would be equivalent to the globbing pattern ["**/*.ts", "!node_modules/**/*.ts", "!tests/**/*.ts"].

[nenadvicentic]

+1

For me, this would do the trick:

{
  ...
  "files": [
    "bower_components/**/dist/*.d.ts",
    "app"
  ],
  "exclude": [
    "node_modules"
  ]
}

[joshheyse]

I'm not sure excludes: [] option is going to able to handle enough scenarios, especially if it is mutually exclusive of files: [].

A lot of our projects have the following structure

├── node_modules
├── src
│   ├── client
│   │   ├── tsconfig.json
│   ├── common
│   ├── server
│   │   ├── tsconfig.json
├── node_modules
├── typings
├── gulpfile.js
├── package.json
├── tsd.json

There are 2 tsconfig files, 1 for the browser and one for node, each with different compile settings and file subsets.

Client tsconfig files: ['**/*.{ts,tsx,js,jsx}', '../common/**/*{ts,tsx,js,jsx}', '../../typings/browser.d.ts'].
Server tsconfig files: ['**/*.{ts,js}', '../common/**/*{ts,js}', '../../typings/main.d.ts'].

The tsconfig in the client directory should include all nested script files, plus the files in common and the browser typings.

The tsconfig in the server directory should include all nested script files, plus the files in common and the main (node) typings.

I haven't been able to come up with a combination of exclude or files (NON GLOB) that won't force me to be continuously managing the files array manually.

Currently we are maintaining the list of file manually. We tried to use atom-typescript to rewire the tsconfig, but it has problems with multiple tsconfig files in a single atom project.

I don't understand why file globing rules can't be used, they can satisfy all the requirements using a standard simplistic rule set.

[glen-84]

@myusrn - Good question. I'll need the same thing. I would be quite surprised if this was not supported.

@nenadvicentic - You would use the include key in future, and the exclude in your case would not be needed.

@joshheyse - You can also use exclude to exclude everything except the files that you need. So for the client you would have something like:

"exclude": [
    "gulpfile.js",
    "node_modules",
    "src/server",
    "typings/main",
    "typings/main.d.ts"
]

A bit less to maintain, but still not ideal.

I don't understand why file globing rules can't be used ...

See my comment above.

[jagt]

I've just started looking into typescript and the most frustrating thing is to get the 'namespaces' to work. Turns out I'll have to list all my files in tsconfig and the namespace resolution order will be correct. If I don't have the tsconfig (I'm using VS2015) or I omit the files section, the namespace will just break and I'll end up with an very stupid error on runtime, caused by the files are concated in random order.

I really like how VS intergrates TypeScript and how the intellisense works almost like wring C#, but the whole thing around namespace and files glob is so irritating and I don't know why there's nobody doing anything with this.

[myusrn]

@jagt i'm thinking that explicitly listing the files you want to compile in tsconfig.json is not the norm and that more often then not one relies on tsconfig.json processing to process all *.ts files found in that path and below it except for excluded entries.

As for namespace resolution I've never run into any issues around ordering. I'm wondering if perhaps the matter here is not having a module loader in place, e.g. systemjs universal module loader, which as i understand it takes cares of loading import referenced namepaces. Once typescript enables use of es6 oob module loader support in, in target: "es5" output, then i'd guess one shouldn't have to enable any module loader in order to get import referenced namespaces loaded in relevant order.

[jagt]

@myusrn I'm sorry about my wording of my last post. It really gets me after searching here and there trying to resolve this weird issue. I'd give you a detailed walk-through of what I've encountered.

As I stated before I've just started looking into TypeScript and started a VS2015 TS project. The first thing I tried to do, after going through syntax and stuff, is to separate code into multiple files. I tried using Modules first, and noticed that it needs an third party module loader like require.js. I'm not that into front end and don't have interest into loader things, and I found out there's this thing called Namespace, which seems to fit my needs exactly, as all I want to do is simply separate my source into multiple files. I do what the doc says, separate my test Class into an other file, and import it from my entry file. It compiles fine but fails in the browser. I checked the compiled output and figured it put the generated class after the first import statement, which is obviously wrong.

After a while I figured I have two solutions, one is to add <reference> comments, which I suppose is deprecated, and the other one to make it work is to manually list all my files in the tsconfig.json. If I do any of this the project will just work, and I'm sticking to the later one.

To make a long story short, I'm trying to find something like browserify in TS and figured that I can do that with namespace and listing files in tsconfig.json, which is kind of confusing.

Again thanks for your kind reply. I really like TypeScript and its VS integration, and despite of this issue (I had worse experience with other technologies 😉 ) most things goes where.

[myusrn]

@jagt i haven't heard anything about use of /// <reference path="<path to .d.ts>" being deprecated and in fact you can see that's what i ended up using in https://github.com/myusrn/acxwithng2 and https://github.com/myusrn/dnxwithng2 main.ts files to allow me to use tsconfig.json with just excludes and still retain support for d.ts sources that exist in some of the excluded folders.

[ghost]

@myusrn +1

[lifenautjoe]

+1, its already a year later 😞

[trickpattyFH20]

+1

[glen-84]

@trickpattyFH20 Please use reactions now, instead of +1 comments.

[dmagoo]

This would also be useful for avoiding ENOENT errors when using an editor that writes and removes backup files fairly often. I find it tripping over /.#file.ts which is often used by emacs.

[alexdima]

For anyone interested in a workaround until filesGlob is added to tsconfig.json:

I've published typescript-with-globs that you can install side-by-side with typescript and you can begin using filesGlob in tsconfig.json.

After installing, you can use tscg (note the extra g) and it will first resolve the globs from filesGlob to the files section and then proceed to invoke tsc.

[mchambaud]

@alexandrudima 👍

[glen-84]

Shouldn't this be added to the TypeScript 2.0 milestone (along with the PRs)?

[mhegazy]

should be addressed by #8841.

[IAMtheIAM]

Great, in case anyone wonder, the property for including files as well as globs is "include", like this

    "include": [
        "./app/**/*.ts"
    ],