please clarify language service import/reference resolution and set-of-files handling

[clausreinke]

I'm in the process of updating a language service client and re-provider (provides access to language services from the commandline, for editor plugins) from codeplex to github versions of typescript, and once again I'm struggling with the sparse documentation.

The variety of interfaces is confusing. One the one hand, there are the compiler vs language service differences, eg, CompilerHost vs ServiceHost, Program vs DocumentRegistry, on the other hand, there seem to be way too many ways of storing sets of file (with associated information).

https://github.com/Microsoft/TypeScript/wiki/Using-the-Compiler-API helps a little bit, but inline docs and a concept map would be helpful, explaining why each of those is necessary and how they relate to each other.

It would be great to have a place to discuss language services API design and usage.

My first concrete question is about import/reference resolution: how do I integrate that with a language service client, without duplicating code?

There used to be a Resolver in the compiler sources that could be used to do nothing but close a set of filenames under import, reference, and filename normalization. That was easy to use.

Now, that functionality seems to be hidden in createProgram. createLanguageService internally uses createProgram, without exposing the Program, but it seems to have no less than four sources of truth about files in the project: the documentRegistry, which seems to be an externally visible copy of the internal sourceFilesByName, in addition to which there are host and hostCache (and program, which is read-only-connected to sourceFilesByName via its compilerHost).

That leaves me with two problems:

1. after createProgram, program.getSourceFiles might have more files than sourceFilesByName (added through import/reference resolution) and I don't see how this information is propagated?

   if I get this right, synchronizeHostData should nearly always do unnecessary work, and will not bring program.getSourceFiles in sync with sourceFilesByName; the language service API functions then use program.getSourceFiles, which is a superset of sourceFilesByName, which should be correct if possibly inefficient (since each of them calls synchronizeHostData)

2. once I have a language service, how do I modify its set of files, taking import/reference resolution into account, without recreating the whole thing from scratch?

[danquirk]

tagging @mhegazy @CyrusNajmabadi @vladima

[basarat]

once I have a language service, how do I modify its set of files, taking import/reference resolution into account, without recreating the whole thing from scratch

I had to do something faily dodgy for https://github.com/TypeStrong/atom-typescript. I plan to move this out of something that depends on languageService and into a simple regex parser that does it as a part of tsconfig's file expansion.

Here is my current code https://github.com/TypeStrong/atom-typescript/blob/ccf844973e946199acf3110acb0923b305051644/lib/main/lang/programManager.ts#L101-L140

increaseProjectForReferenceAndImports = () => {

        var willNeedMoreAnalysis = (file: string) => {
            if (!this.languageServiceHost.hasScript(file)) {
                this.languageServiceHost.addScript(file, fs.readFileSync(file).toString());
                this.projectFile.project.files.push(file);
                return true;
            } else {
                return false;
            }
        }

        var more = this.getReferencedOrImportedFiles(this.projectFile.project.files)
            .filter(willNeedMoreAnalysis);
        while (more.length) {
            more = this.getReferencedOrImportedFiles(this.projectFile.project.files)
                .filter(willNeedMoreAnalysis);
        }
    }

    getReferencedOrImportedFiles = (files: string[]): string[]=> {
        var referenced: string[][] = [];

        files.forEach(file => {
            var preProcessedFileInfo = ts.preProcessFile(this.languageServiceHost.getScriptContent(file), true),
                dir = path.dirname(file);

            referenced.push(
                preProcessedFileInfo.referencedFiles.map(fileReference => utils.pathResolve(dir, fileReference.filename))
                    .concat(
                    preProcessedFileInfo.importedFiles
                        .filter((fileReference) => utils.pathIsRelative(fileReference.filename))
                        .map(fileReference => utils.pathResolve(dir, fileReference.filename + '.ts'))
                    )
                );
        });

        return utils.selectMany(referenced);
    }

And you simply call increaseProjectForReferenceAndImports after you have created your langaugeService:

        this.languageServiceHost = new languageServiceHost.LanguageServiceHost(projectFile);
        this.languageService = ts.createLanguageService(this.languageServiceHost, ts.createDocumentRegistry());

        // Now using the language service we need to get all the referenced files and add them back to the project
        this.increaseProjectForReferenceAndImports();

FYI the magic function getReferencedOrImportedFiles came from @fdecampredon's project services : https://github.com/TypeStrong/typescript-project-services

[basarat]

without recreating the whole thing from scratch

It would be great if I didn't need to call ts.preProcessFile(this.languageServiceHost.getScriptContent(file), true)

[clausreinke]

For now, I've put the project file information into (service) host (as mutable arrays in closures), hoping that hostCache won't interfere. This looks similar to what you do, only that I haven't yet accounted for the new project files.

Also, I create a throwaway Program before creating the LanguageService, so that I have a fully resolved set of filenames at hand (this looks like a waste, though, just like your use of preProcessFile to get a hand on the implied files; but as long as the project file list is exposed read-only from the service host to createLanguageService, and the internal program isn't exposed at all, something like this seems to be necessary).

I still have no idea what documentRegistry is good for - exposing the program from createLanguageService might be more useful?

[basarat]

exposing the program from createLanguageService might be more useful

Potentially, considering that program has a public API. I suspect they haven't done it because they want the LanguageService users to have a consistent meaning of file (which is different from the meaning of file used by program).

[basarat]

Basically I think you are either in languageService world or program world depending upon your goal.

[mhegazy]

First sorry for the late reply, I needed to write down a brain dump of some of the LS information so that i can share it.
Second, sorry for the lack of documentation for the Language service and the API in general; we are currently working on adding these as we go. your feedback is externally important. so feel free to flag areas that are ambiguous or that need more documentation.

I have added a new Language Service API page for details about the API. i have also cleaned up the Architecture Overview. take a look and let me know if you need more information.

Just like @basarat explained, the gist is that the language service has a different interaction model than the compiler when it comes to resolution to allow for a complete control to the host. that means you will need to do it in a two-step process, first resolve the context using preprocessFile to know what files are referenced then return them to the language service object when it asks for getScriptFileNames.

@clausreinke we understand that there are functionality that most editors would need to implement on top of the language service API, and that the current API by itself is not sufficient. We are working on isolating these into a new wrapper that can ease editor plugin creation. I would want that as part of the main TypeScript offering. It would be great if we can cooperate on it.

[basarat]

@mhegazy Excellent! I didn't know the reason for document registry. Super cool!

[clausreinke]

@mhegazy Thanks, that's just the kind of documentation I was hoping for! And overall, the API has gotten easier to use (modules, finally, yay!-).

(to prevent it going stale later, it would be good to include the relevant bits in the sources, as comments that make it through to the declaration file, and hence intellisense; one can still generate wiki pages from in-source comments;-)

While it is not difficult to create a resolver, there are many conventions encoded in that bit of code, and duplicating such code risks copies going out of sync (eg, when TypeScript moves to ES6 modules). It should be possible to provide an API that encodes the conventions, without interfering with the two interaction models.

I still think there are problems in the design, similar to point 1 above. In case I don't get around to detailing them later, here are some more quick notes:

• in spite of the different interaction models, services are an utility layer on top of the compiler API
• not exposing the internal compiler use risks duplicating code when compiler and service usages are combined on the same project
• createLanguageService takes a service host, but creates its internal program with a default compiler host - that can't be right unless the service host is a default host, too, right?
• shouldn't it be possible to create one host from the other (probably service from compiler, by hiding some parts and adding others)?
• the entirely pull-based design of service/compiler creation wrt to service/compiler hosts means that service/compiler can't push back information to their hosts, even though those currently hold the project data; as in the example of host holding a project root file and program resolving imports, but not updating any implicit host project file list (I also noted that SourceFiles keep their unresolved file names, instead of canonical versions thereof)

[mhegazy]

(to prevent it going stale later, it would be good to include the relevant bits in the sources, as comments that make it through to the declaration file, and hence intellisense; one can still generate wiki pages from in-source comments;-)

@DanielRosenwasser is working on adding documentation to the public interfaces. we should have that by next release. PRs are always welcomed :)

• in spite of the different interaction models, services are an utility layer on top of the compiler API

Yes, however, a singe LanguageService object does not map to a single Program object, it actually maps to a set of them. A Program is an immutable object, the LanguageService on the other hand handles changes. under the covers, the LanguageService, on every change, creates a new Program instance to map to the current context, but uses the previous Program instance's unchanged SourceFiles to save computation. If the LanguageService was just an extension over Program, you would need to create a new LanguageService object on every change.

• not exposing the internal compiler use risks duplicating code when compiler and service usages are combined on the same project

That is correct. I am open to making the language service resolve the context and expose a getProgram method. but that does not simplify the process. The host is still responsible for managing the files.
Consider this case, host knows about only one file, it does not do resolution, it just passes it to the language service, the LanguageService figures out the context, but the host still need to monitor the other files in the context that were referenced by the main files. The LanguageService does not monitor files or interact with the underlying system in any way. so now the host needs to ask the language service after every change about the "complete" list of files in the context to add watches to them.

• createLanguageService takes a service host, but creates its internal program with a default compiler host - that can't be right unless the service host is a default host, too, right?
• shouldn't it be possible to create one host from the other (probably service from compiler, by hiding some parts and adding others)?

it does not create a default compiler host, it creates a wrapper around its own host. as there are a bit of differences. i agree that a LanguageServiceHost that is a subtype of CompilerHost would be ideal. let me see what i can do here.

• the entirely pull-based design of service/compiler creation wrt to service/compiler hosts means that service/compiler can't push back information to their hosts, even though those currently hold the project data; as in the example of host holding a project root file and program resolving imports, but not updating any implicit host project file list (I also noted that SourceFiles keep their unresolved file names, instead of canonical versions thereof)

The idea is that we do not amuse any thing about the underlying system. We have host implementation that do resolution totally different as they have environment. the language service still handles that. that is the same reason the sourceFile stores the data, but does not attempt to "resolve" them. the caller can decide what to do with them. We need to add a "resolveImport" helper that allows you to take a compilerHost/LanguageServiceHost and a module name, and return a full file path. so far our import resolution is fairly simple, but once we through in tsconfig, and node_module it is not going to be mangable. we should have that soon though.

[clausreinke]

under the covers, the LanguageService, on every change, creates a new Program instance to map to the current context, but uses the previous Program instance's unchanged SourceFiles to save computation.

In part, that explains why it is considered cheap to (potentially) recreate the program on every API call. It does, however, remind me of a standard problem with transitive import systems: change can propagate through unchanged files: so if the traversal is simply cut of at the top-most unchanged file in the tree, changes propagating from below may be missed. I haven't yet checked the code to verify whether this is a problem here as well, but from your description it might well be.