Language service proxy #12231

RyanCavanaugh · 2016-11-14T21:03:10Z

Implements basic language service proxy plugins

RyanCavanaugh · 2016-11-14T21:07:50Z

weswigham · 2016-11-15T03:59:10Z

src/server/project.ts

@@ -142,6 +154,35 @@ namespace ts.server {
            return this.cachedUnresolvedImportsPerFile;
        }

+        public static resolveModule(moduleName: string, initialDir: string, host: ServerHost, log: (message: string) => void): {} {


Why not just reuse the module resolver the TS compiler uses elsewhere? I understand that the module resolver is a lot more complicated thanks to its exposure the the compiler options, but it should be configurable to load just JS, right? At least, it'd probably be better not to duplicate the folder traversal/lookup logic in two places?

I considered this but didn't think it was worthwhile given that we'd have to mock out a CompilerOptions guaranteed to behave the same as NodeJS's resolution (which still differs from our own given how we crack open package.jsons and other stuff). I'm actually going to switch this to just use require directly anyway now that I've learned how to set up a different initial search path.

@RyanCavanaugh Shouldn't require at least be abstracted behind a call to sys?

@weswigham I don't really see a point - is anyone plausibly going to write a plugin that works under a host that's not node?

what about substituting require in unit tests?

And I'm sure someone like @basart would like to support angular and other plugins in his cloud IDE.

I don't really see a point - is anyone plausibly going to write a plugin that works under a host that's not node?

@RyanCavanaugh Isn't there an intent for this to work in Visual Studio? Our Chakra host would need to implement its own require logic in that case.

@DanielRosenwasser VS17 uses tsserver + node so it won't be a problem

It's already been refactored into sys as suggested.

weswigham · 2016-11-15T04:01:14Z

src/server/project.ts

+                return;
+            }
+
+            if (typeof require === "undefined") {


Rather than using the global require directly, shouldn't this be abstracted as a host detail under sys?

weswigham · 2016-11-15T04:03:20Z

src/server/project.ts

+            const items: string[] = [];
+            for (const plugin of this.plugins) {
+                if (plugin.getExternalFiles) {
+                    items.push(...plugin.getExternalFiles(this));


If I'm not mistaken, this calls plugin code which can throw - any errors should probably be handled and logged?

Yup. we need to wrapp this in a try..catch block to make sure the project is not in an invalid state.

weswigham · 2016-11-15T04:26:48Z

src/harness/fourslash.ts

+                    const proxy = Object.create(null);
+                    const langSvc: any = info.languageService;
+                    for (const k of Object.keys(langSvc)) {
+                        proxy[k] = function () {


The LS codebase (and the API's consumers) are not resilient to exceptions being thrown from these API methods, however consumers of the plugin API are not bound by any contract forbidding exceptions. Is there some way we can force a safer construction of these proxy methods whereby execution errors caused by improper plugins are appropriately logged and recorded, rather than surfaced as ICEs?

weswigham · 2016-11-15T04:27:55Z

src/server/project.ts

@@ -90,6 +90,18 @@ namespace ts.server {
        }
    }

+    export interface PluginCreateInfo {


I'm not sure if this is still a priority, but should plugins be given the opportunity to be well-behaved with respect to a cancellation token?

+1, plugins should be monitoring cancellation requests similar to normal LS methods

RyanCavanaugh · 2016-12-29T00:11:56Z

@mhegazy ping

RyanCavanaugh · 2016-12-30T00:37:57Z

Angular-side PR is up at angular/angular#13716

RyanCavanaugh · 2017-01-07T00:57:28Z

Ping anyone...

RyanCavanaugh · 2017-01-11T03:24:22Z

@mhegazy @vladima ?

RyanCavanaugh · 2017-01-17T23:18:51Z

Please?

billti · 2017-01-18T00:31:18Z

src/compiler/sys.ts

+
+                    // Note: 'require' needs the path to be all forward slashes and absolute
+                    initialDir = normalizeSlashes(nodeSystem.resolvePath(initialDir));
+                    mod.paths.unshift(initialDir);


I'd be wary of this. 'module.paths' isn't documented at all on the Node.js site (https://nodejs.org/dist/latest-v6.x/docs/api/modules.html). I'd assume this is an internal property which could change or disappear at any time.

Definitely aware (see earlier comment in this function for a link to an SO question about this). I considered this to be preferable to re-implementing their module resolution algorithm since we don't necessarily know how it works in all cases. Thoughts?

billti · 2017-01-18T00:35:46Z

src/compiler/types.ts

@@ -3358,7 +3364,7 @@ namespace ts {
    /* @internal */
    export interface CommandLineOptionOfListType extends CommandLineOptionBase {
        type: "list";
-        element: CommandLineOptionOfCustomType | CommandLineOptionOfPrimitiveType;
+        element: CommandLineOptionOfCustomType | CommandLineOptionOfPrimitiveType | TsConfigOnlyOption;


This reads weird and needs a comment if deliberate. By definition, I would assume a "TsConfigOnlyOption" can't be provided as a CommandLineOption?

It's how everything here is named. Note that CommandLineOptionBase has an isTSConfigOnly property, which you would think would have to be false by definition

billti · 2017-01-18T00:47:49Z

src/server/project.ts

+            }
+
+            for (const pluginConfigEntry of options.plugins) {
+                const searchPath = combinePaths(this.configFileName, "..");


This reads like it's going up a folder to the parent of the folder with the config file, but I assume the intent is just to remove the tsconfig.json and leave the folder. Looking at the implementation of combinePaths, I believe it's just going to result in these being concatenated with a directory separator (e.g. /src/project/tsconfig.json/..), which may work for 'resolveModule', but seems a little bizarre.

Will change to path.dirname

getDirectoryPath rather

billti · 2017-01-18T00:53:28Z

src/server/project.ts

+            }
+            catch (e) {
+                this.projectService.logger.info(`Plugin activation failed: ${e}`);
+                this.projectService.logger.info(`Plugin object was: ${pluginModule}`);


Isn't this just going to log as [object]? Did you mean to JSON.stringify it?

If it's an object, yes. In this case I needed to find out that it wasn't null, wasn't undefined, wasn't a class, wasn't a primitive, and wasn't a function (the angular module was returning a factory function when I expected an object in this case). I would be worried calling JSON.stringify here as it could trigger a secondary exception during a property getter

You might want to browse the spec (http://es5.github.io/#x15.12.3) and run some testing to verify, but JSON.stringify never throws (even for undefined, null, NaN, etc..) and usually outputs what you would want.

It does throw.

If stack contains value then throw a TypeError exception because the structure is cyclical.

Right, I should have been clearer. I meant based on the property type (i.e. addressing the comment I needed to find out that it wasn't null, wasn't undefined, wasn't a class, wasn't a primitive, and wasn't a function).

Updated to do some better logging of Objects. I believe Object.keys cannot throw on an object input

billti · 2017-01-18T01:00:30Z

tests/cases/fourslash/server/ngProxy1.ts

+//// 
+
+goTo.marker();
+verify.quickInfoIs('Proxied x: number[]Check');


The tests seem a little light for a new feature. We need to test what happens when the plugin fails to load (either throws on initialization or can't be found), if the plugin config is misconfigured, if the proxied API returns invalid results, etc... We need to ensure we're either resilient or give an actionable error to help identify the cause, else we'll be troubleshooting a lot of issues that aren't ours 😟

skreborn · 2017-02-08T14:18:28Z

Hate to be bumping, but is there anything that still prevents this from being ready?

prabirshrestha · 2017-02-09T19:56:40Z

How does plugins work under watch mode?

For example if I create a plugin to require resx file for resources, I would like to automatically watch all languages besides the imported file and generate a typescript equivalent code.
This also means if some stops importing the resx file I need to stop watching.

mhegazy · 2017-02-09T22:00:44Z

src/server/project.ts

+        public static resolveModule(moduleName: string, initialDir: string, host: ServerHost, log: (message: string) => void): {} {
+            const resolvedPath = normalizeSlashes(host.resolvePath(combinePaths(initialDir, "node_modules")));
+            log(`Loading ${moduleName} from ${initialDir} (resolved to ${resolvedPath})`);
+            const result = host.require(resolvedPath, moduleName);


i would wrap this in a try/catch block and log the exception

or just do this in enableProxy

require can't throw. It returns a module or an error.

can you please put it as a comment to ServerHost.require since we don't have any other ways to declare/enforce this contract?

mhegazy · 2017-02-09T22:06:36Z

src/server/project.ts

+                    languageServiceHost: this.lsHost,
+                    serverHost: this.projectService.host
+                };
+                if (pluginModule.create === undefined && typeof pluginModule === "function") {


is there a reason why we have two ways to define a module? why would not create take ts as well? how would it get ts?

Can we define the shape of a plugin in an interface as well.

This is how angular's module loading scheme works. We could force everyone to follow this pattern, or allow both ways. Thoughts?

mhegazy · 2017-02-09T22:07:45Z

src/server/project.ts

+        getExternalFiles(): string[] {
+            const items: string[] = [];
+            for (const plugin of this.plugins) {
+                if (plugin.getExternalFiles) {


typeof plugin.getExternalFiles !== "function"

mhegazy · 2017-02-09T22:08:28Z

src/server/project.ts

+            const items: string[] = [];
+            for (const plugin of this.plugins) {
+                if (plugin.getExternalFiles) {
+                    items.push(...plugin.getExternalFiles(this));


Yup. we need to wrapp this in a try..catch block to make sure the project is not in an invalid state.

vladima · 2017-02-09T22:15:28Z

scripts/buildProtocol.ts

@@ -167,7 +167,7 @@ function generateProtocolFile(protocolTs: string, typeScriptServicesDts: string)
    const sanityCheckProgram = getProgramWithProtocolText(protocolDts, /*includeTypeScriptServices*/ false);
    const diagnostics = [...sanityCheckProgram.getSyntacticDiagnostics(), ...sanityCheckProgram.getSemanticDiagnostics(), ...sanityCheckProgram.getGlobalDiagnostics()];
    if (diagnostics.length) {
-        const flattenedDiagnostics = diagnostics.map(d => ts.flattenDiagnosticMessageText(d.messageText, "\n")).join("\n");
+        const flattenedDiagnostics = diagnostics.map(d => ts.flattenDiagnosticMessageText(d.messageText, "\n") + ' at ' + d.file.fileName + ' line ' + d.start).join("\n");


nit:

`${ts.flattenDiagnosticMessageText(d.messageText, "\n")} at ${d.file.fileName} line ${d.start}

vladima · 2017-02-09T22:29:42Z

src/compiler/sys.ts

@@ -569,7 +573,31 @@ namespace ts {
                    }
                },
                setTimeout,
-                clearTimeout
+                clearTimeout,
+                require(initialDir: string, moduleName: string) {


can we move require from sys to ServerHost?

it looks like currently only tsserver will use it

if we do it in tsserver we can use our module resolution implementation and then do require with resolved path without relying on undocumented properties

mhegazy · 2017-02-14T01:33:31Z

src/compiler/sys.ts

@@ -45,6 +45,8 @@ namespace ts {
        clearTimeout?(timeoutId: any): void;
    }

+    export type RequireResult = { module: {}, error: undefined } | { module: undefined, error: {} };


I would move this to types.ts or server.ts

RyanCavanaugh · 2017-02-14T18:38:05Z

@mhegazy any other comments?

mhegazy · 2017-02-14T19:33:26Z

Looks good to me. @vladima any final comments?

vladima · 2017-02-14T21:25:34Z

src/compiler/moduleNameResolver.ts

@@ -683,14 +683,14 @@ namespace ts {
        }
    }

-    export function nodeModuleNameResolver(moduleName: string, containingFile: string, compilerOptions: CompilerOptions, host: ModuleResolutionHost, cache?: ModuleResolutionCache): ResolvedModuleWithFailedLookupLocations {
+    export function nodeModuleNameResolver(moduleName: string, containingFile: string, compilerOptions: CompilerOptions, host: ModuleResolutionHost, cache?: ModuleResolutionCache, jsOnly = false): ResolvedModuleWithFailedLookupLocations {


nit: can we instead of using default parameter split this function into two:

export function nodeModuleNameResolver(moduleName: string, containingFile: string, compilerOptions: CompilerOptions, host: ModuleResolutionHost): ResolvedModuleWithFailedLookupLocations { return nodeModuleNameResolverWorker(moduleName, containingFile, compilerOptions, host, /*cache*/ undefined, /*jsOnly*/ false); } /* @internal */ export function nodeModuleNameResolverWorker(moduleName: string, containingFile: string, compilerOptions: CompilerOptions, host: ModuleResolutionHost, cache: ModuleResolutionCache, jsOnly): ResolvedModuleWithFailedLookupLocations { ... }

and make internal callers use nodeModuleNameResolverWorker instead of nodeModuleNameResolver
Reason: I think it is preferable to keep public interface of the compiler clean and not pollute it with internal intricacies

vladima · 2017-02-14T21:26:31Z

src/compiler/sys.ts

@@ -582,7 +582,7 @@ namespace ts {
                    }
                },
                setTimeout,
-                clearTimeout
+                clearTimeout,


nit: extra comma

vladima · 2017-02-14T21:29:31Z

src/server/project.ts


        public languageServiceEnabled = true;

+        protected lsHost: LSHost;


can you also make it readonly? AFAIR nobody should change it once LSHost is created

vladima · 2017-02-14T21:32:29Z

src/server/project.ts

+                    serverHost: this.projectService.host
+                };
+
+                if (typeof pluginModuleFactory !== "function") {


move the check before making info

vladima

LGTM, with a nit about making plugin aware of cancellation token

DanielRosenwasser · 2017-02-14T22:49:09Z

🎉

mihailik · 2017-02-15T18:34:12Z

Is this what's described in #11976 ?

I can see that @angelozerr says in #13436 (comment) thanks @mhegazy for your answer. Now I see how to plugins to load are declared (in the tsconfig.json).

But #13436 doesn't actually include any answer with such an example. Was the usage described elsewhere? Offline?

Design notes mention this PR only in passing too.

It looks like a huge layer of functionality implemented on the sly! 😎 ㊙️

mhegazy · 2017-02-15T19:32:17Z

We still need to add documentation for this feature. This is scheduled for TS 2.3 at the moment.

guncha · 2017-04-21T15:29:24Z

With these changes in place, can we already achieve the holy grail of extensions and introduce arbitrary types in the system or is that still some way off? There are two use-cases that I'm thinking of:

const foo = Relay.QL`fragment on Foo { name, bar { count } }`
require.ensure("lib/bar", (error, bar) => bar.baz())

In the first case, the rest of the code should be type checked as if foo had a type { value: {name: string, bar: { count: number }}} and the second is a way to code-split with webpack and the bar in the callback should be typed as the module from "lib/bar".

The type checker in checker.ts is very encapsulated so there's no real way to extend any of the methods. Some of this could be accomplished by changing the SourceFile right after it's parsed, but that seems a little hacky.

mhegazy · 2017-04-21T16:26:19Z

@guncha, this is just an LS extension, it does not change any of the types in checking phase. i believe you are looking for a different extension point, something like #3136

msftclas added the cla-not-required label Nov 14, 2016

weswigham reviewed Nov 15, 2016

View reviewed changes

RyanCavanaugh force-pushed the ls-proxy branch from 1c52714 to d0f3dc7 Compare November 22, 2016 22:48

mhegazy mentioned this pull request Dec 2, 2016

Design Meeting Notes, 11/30/2016 #12622

Closed

RyanCavanaugh mentioned this pull request Dec 30, 2016

Support TypeScript plugin extensibility model for Language Service features angular/angular#13716

Closed

3 tasks

mhegazy mentioned this pull request Jan 12, 2017

How to implement TypeScript 2.2 plugin? #13436

Closed

billti reviewed Jan 18, 2017

View reviewed changes

billti suggested changes Jan 18, 2017

View reviewed changes

Quramy mentioned this pull request Feb 8, 2017

Better integration Quramy/ng-tsserver#1

Closed

mhegazy suggested changes Feb 9, 2017

View reviewed changes

vladima reviewed Feb 9, 2017

View reviewed changes

mhegazy reviewed Feb 14, 2017

View reviewed changes

mhegazy approved these changes Feb 14, 2017

View reviewed changes

vladima reviewed Feb 14, 2017

View reviewed changes

vladima approved these changes Feb 14, 2017

View reviewed changes

Language service extensibility

aec3109

RyanCavanaugh force-pushed the ls-proxy branch from 188fb76 to aec3109 Compare February 14, 2017 21:36

Fix downstream use

86cc405

RyanCavanaugh merged commit 87b780d into microsoft:master Feb 14, 2017

RyanCavanaugh deleted the ls-proxy branch February 15, 2017 18:38

HerringtonDarkholme mentioned this pull request Feb 17, 2017

TypeScript language service can't find name HerringtonDarkholme/av-ts#44

Closed

orta mentioned this pull request Feb 24, 2017

TypeScript extensibility #6508

Closed

MichaReiser mentioned this pull request Mar 2, 2017

Plugin Support for Custom Transformers #14419

Open

guncha mentioned this pull request Mar 25, 2017

Plugin support s-panferov/awesome-typescript-loader#401

Open

Quramy mentioned this pull request Apr 3, 2017

Move "Language Service Extensibility" microsoft/TypeScript-wiki#114

Closed

microsoft locked and limited conversation to collaborators Jun 19, 2018


		public languageServiceEnabled = true;

		protected lsHost: LSHost;

Language service proxy #12231

Language service proxy #12231

Conversation

RyanCavanaugh commented Nov 14, 2016

RyanCavanaugh commented Nov 14, 2016

weswigham Nov 15, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DanielRosenwasser Dec 29, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RyanCavanaugh commented Dec 29, 2016

RyanCavanaugh commented Dec 30, 2016

RyanCavanaugh commented Jan 7, 2017

RyanCavanaugh commented Jan 11, 2017

RyanCavanaugh commented Jan 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gcnew Jan 19, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skreborn commented Feb 8, 2017

prabirshrestha commented Feb 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladima Feb 9, 2017 • edited Loading

Choose a reason for hiding this comment

mhegazy Feb 14, 2017 • edited Loading

Choose a reason for hiding this comment

RyanCavanaugh commented Feb 14, 2017

mhegazy commented Feb 14, 2017

vladima Feb 14, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladima left a comment

Choose a reason for hiding this comment

DanielRosenwasser commented Feb 14, 2017

mihailik commented Feb 15, 2017 • edited Loading

mhegazy commented Feb 15, 2017

guncha commented Apr 21, 2017

mhegazy commented Apr 21, 2017

weswigham Nov 15, 2016 •

edited

Loading

DanielRosenwasser Dec 29, 2016 •

edited

Loading

gcnew Jan 19, 2017 •

edited

Loading

vladima Feb 9, 2017 •

edited

Loading

mhegazy Feb 14, 2017 •

edited

Loading

vladima Feb 14, 2017 •

edited

Loading

mihailik commented Feb 15, 2017 •

edited

Loading