Expose incremental Compile-on-Save builder API

[DanielRosenwasser]

In #10879, @sheetalkamat is working on bringing the same logic in the language server to Compile-on-Save. We would like to expose that as an API so that other build tools (e.g. Webpack loaders, gulp plugins, Broccoli plugins) and editors (any editor using TS Server) can get:

• automated resolution logic to detect which files need to be "updated"
• incremental type-checks to only check the bare minimum of files
  • this means that, for example, requesting errors & building will only trigger a type check on updated/new/impacted files. By "impacted files", I mean files whose type information could potentially have been changed by other updated/new files.

Some prior work that motivated the scenarios here can be seen on PR #3687 by @dbaeumer.

We'd like to elicit feedback from others in the community. Some folks that immediately come to mind are @TheLarkInn @johnnyreilly @jbrantly @s-panferov @mattbierner @jrieken @dbaeumer @alexeagle @chuckjaz @ivogabe and @krisselden (Forgive me if I excluded anyone, that's already a decent number of people 😄).

[johnnyreilly]

Exciting!! cc @piotr-oles @basarat @blakeembrey @HerringtonDarkholme @MortenHoustonLudvigsen

It sounds like the addition of this will make for simpler build tools. Put simply, I figure certain tools will need to do less work with this is in place. Perhaps it would be useful for people to list "things their tools do" which, with the addition of this functionality, they think they won't need to anymore.

If we cover that we will hopefully ensure that what lands will work in the way that everyone hopes.

@jbrantly - I know you're less involved with ts-loader these days but whenever you pitch in your insights are really useful. If you could get a moment to comment on this I think it would be really valuable for everyone. (Obviously no worries if you can't)
❤️ 🌻

[TheLarkInn]

Will this be an api that has both sync and async variants?

Additionally, this sounds awesome so far. I think this lets both @johnnyreilly and @s-panferov remove the soon-to-be deprecated accessing of tapable instances inside the loaders. (Which means tools like happypack @hansl and friends can use @ngtools/webpack maybe as a loader instead (if not much has changed there). Just many many opportunities are now possible, etc.

💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯 💯

[johnnyreilly]

Hi @TheLarkInn,

soon-to-be deprecated accessing of tapable instances inside the loaders

Do you have a link you could share to this please Sean? I hadn't heard this was happening and would like to read up on it. Thanks!

Which means tools like happypack @hansl and friends can use maybe as a loader instead

As an aside, you can use ts-loader with happypack right now. It essentially works by flipping ts-loader into transpile mode and preventing reporting of errors to webpack in the way that ts-loader normally does. (Type checking is moved into another process via a plugin.)

The reason I mention that is the preventing of reporting to webpack etc is I suspect where we make use of tapable instances. I just want to be clear that I understand where we're hooking into this if that's going to need to change. Thanks!

[DanielRosenwasser]

To answer the question about sync vs async, I don't think we will have an async API since the compiler runs synchronously.

Otherwise, something that would be helpful for @sheetalkamat to know is what sort of API you all expect to be able to use. Do we have users who need to be able to query for only new diagnostics from files, or is it whole-program errors every time?

Currently the stripped down design is something like the following

createWatchMode(
    sysHost: System, 
    callback: (program: Program, changedFiles: string[]) => void,
): void

With that, a callback could potentially always just query the program for when it needs new diagnostics, or needs to emit those specific files. It could be that consumers are responsible for caching diagnostics themselves.

[krisselden]

@DanielRosenwasser for additions/removals would one diff the prior program source files to the program passed to the callback?

[sheetalkamat]

@krisselden The changedFiles is list of addition/removed/changed files name list.

[johnnyreilly]

Do we have users who need to be able to query for only new diagnostics from files, or is it whole-program errors every time? .... It could be that consumers are responsible for caching diagnostics themselves.

My feeling is that having "whole-program errors every time" is simpler from a consumption perspective. Less scope for getting things wrong, less work to do (selfish I know!)

Looking at the API, I'm assuming the idea is you'd call the callback with changed files and then query for diagnostics immediately after? And this would be fine as the API is sync not async?

Obviously you're only at design stages right now but if you have a chance, I'd love to see some simple code examples of how you would expect this API to be used. Personally, I find it easier to think about things when I see examples of how something plugs together (and also I've found that when I take things out of the abstract to the practical it shows up considerations I hadn't pondered.) 🌻

[DanielRosenwasser]

Also I guess this is #9282, sorry @alexeagle.

[filipesilva]

I'd also like to see a usage example for the proposed createWatchMode.

[masaeedu]

This is interesting. Could you provide some more details on what features this compile-on-save API exposes? Is it a "read-only" API that simply informs external services of TypeScript's dependency model, or can it also be used to inject other tasks into the dependency graph? Something like the latter would be pretty useful for doing codegen.

[csvn]

With the current Compiler API, a custom CompilerHost can be provided with adjusted methods for e.g. getSourceFile. Will this proposed createWatchMode API allow such modifications (such as for createProgram)? I haven't worked on any open source tool, so I don't know if that would be required. Though it would be useful for an internal watch script I made recently with the Compiler API.

[ikokostya]

It will be great if command line interface of TypeScript compiler (tsc) will also support incremental building. This mode allows embed tsc in command pipeline much easier, e.g.

# Compile and run tests
tsc -p tests --outDir tests-out && mocha tests-out

Now this command has two disadvantages:

1. need wait for full rebuild of the project described by tests/tsconfig.json;
2. or need keep watch mode (tsc --watch) as separate process.

[kenotron]

@ikokostya there are projects like ts-watch that allows you to do watch + trigger onSuccess.

[OneCyrus]

is the builder API still on track for 2.7?

[gilamran]

@ikokostya for now you can use tsc-watch to re-run a task on successful build

[mhegazy]

Should be fixed by #20234