What is the API story for this new codebase?

[DanielRosenwasser]

How will the API work? Will developers have a canonical JavaScript-based API for integrating with this new version of TypeScript?

---

While we are porting most of the existing TypeScript compiler and language service, that does not necessarily mean that all APIs will be ported over. Because of the challenges between language runtime interoperability, API consumers will typically not communicate within the same process. Instead, we expect our API to leverage a message-passing scheme, typically over an IPC layer. Because this sort of bridging is not "free", exposing all functionality will be impractical. We expect to have a more curated API that is informed by critical use-cases (e.g. linting, transforms, resolution behavior, language service embedding, etc.).

We knew that providing a solid API would be a big challenge, so as soon as we started exploring the TypeScript port, we investigated the possibilities here. Beyond how capable the API would be, we asked ourselves whether the sorts of use-cases would be constrained by the performance of IPC solutions. More specifically, even if we could come up with a set of APIs that did what users wanted, would the throughput of requests be a limiting factor? Would the new TypeScript be fast enough to offset the cost of serialization and data transfer? Would the chattiness of API usage overwhelm the wins of having a much faster compiler?

We've become increasingly confident and optimistic in answers around the IPC performance. @zkat has built a Node native module to use synchronous communication over standard I/O between external processes. Building on that, @andrewbranch has experimented with exposing an API server entrypoint to our compiler, along with a JavaScript client that "speaks" to the server over that communication layer. What we've found is fairly promising - while IPC overhead is not entirely negligible, it is small enough. We also can imagine opportunities to optimize, use other underlying IPC strategies, and provide batch-style APIs to minimize call overhead.

As our experiments solidify, we will post more concrete details on our plans, and what the API will look like.

[SLaks]

What are your IPC plans for WebAssembly in the browser?

[jakebailey]

I don't think we have a concrete plan (just prototypes), but we expect to be able to at least do the same IPC via mechanisms like WASI (what I do in https://jakebailey.dev/esbuild-playground, not to pump this side project). There's also the possibility to use plain messaging and only offer async in the browser, or some sort of async worker with SharedArrayBuffer based communication; the latter is already pretty well thought out for vscode.dev/github.dev, so there's theoretically some reusability there.

[rictic]

Have you considered WASM as a solution to providing a cross-language API?

I haven't used it for this purpose, but it seems like there could be a good match in requirements. WASM modules can pass typed data to one another, call one another's functions, etc.

[Meligy]

Actually, this has its own discussion here:
#514

[jakebailey]

People really need to provide their test code. I'm pretty sure the results are incorrect. Notably, threading needs to be disabled in WASM but we didn't do that in the repo yet.

[marcus-sa]

Wouldn't https://extism.org be ideal here?

[jakebailey]

Why is exitism ideal? Which problem are you specifically trying to solve? Plugins?

To be clear, I do not think it's the direction we want to take to allow embedding languages into tsc. It could be technically feasible, but we'd now be a "runtime" and have to go through a lot of compliance work.....

[marcus-sa]

@jakebailey it could allow us to hook into the compiler similarly to how its currently being done with the transformer API.

[dummdidumm]

How's the story for TypeScript plugin developers and people integrating with the language service? More specifically, will method patching still work? The official way to create a plugin is to patch methods. And across such use cases there are probably dozens of cases where people reach into TypeScript semi-private internals to patch other methods (so that things can be intercepted for example) - will that also still work? If not, how do you envisage plugin authors to migrate away from doing that?

[dummdidumm]

Yes that is probably my biggest concern - but since this all needs to be redesigned anyway, it sounds like you are also open to add proper hooks into these internals where it's needed. I think the Angular team, the Volar.js team and the Svelte team (which I'm a part of) are all doing lots of stuff with both the language service API and the plugin API, and are probably all very interested to talk details once the internals have stabilized a bit.

[jedwards1211]

@andrewbranch I'm bummed that it won't be possible to use ts-morph anymore to do codemods...people are mostly talking about more basic AST walking, but is type information going to be part of the API that operates over IPC?

[jakebailey]

We love ts-morph; it is explicitly an anti-goal to prevent ts-morph from working altogether. Type information is effectively the reason why we need to have an API at all 😄

[jedwards1211]

Okay whew!

[itsdouges]

WE'RE SO BACK

[saw]

I've used the tsc api for various codegen purposes, will you be exposing an API to walk/visit the AST tree as was available in the js tsc api?

[Stanzilla]

Will error reporting change? For a long time, TS errors have been a heated topic and there have been efforts by the community (https://github.com/yoavbls/pretty-ts-errors) to make it a little better. Will solutions like Pretty TS still work in the new world?

[lucsoft]

Are there plans about instead of IPC there would be something like FFI?

[jakebailey]

FFI with Go is problematic because Go assumes it's the only instance of Go in a process, but people can and do import multiple versions of TypeScript (for example, to test DefinitelyTyped); while we did have some success loading multiple instances in the same process, it's not something the Go team supports.

[jedwards1211]

Is there really no way to bootstrap go's global state once and then instantiate multiple things within that same global state from separate NAPI calls?

[jakebailey]

Loading multiple runtimes into the same process means they are all competing to register signal handlers, do preemption, manage certain process-specific limits, etc. It might work if all are built with the same exact version of Go, since things like signal handlers would still grab data at the right offsets into things like TLS, but it's definitely not a guaranteed.

See also: golang/go#65050 (comment)

[jedwards1211]

What I mean is, I'm surprised it's not possible to only load a single runtime on the first FFI call, and have subsequent FFI calls instantiate structs into that same runtime...

Which from skimming that thread, sounds like it is possible using plugins instead of c-shared libraries...but I guess there's still a problem if different versions of typescript-go are built for different versions of the Go runtime?

[jakebailey]

The Go runtime is entirely different between Go releases, and potentially Go patches. I don't even think it's guaranteed that it'll work if they happen to match.

Plugins are not an option; that's for loading Go code into Go code, not Go code into other things. That and Go plugins do not support Windows.

[puchm]

Not a question, but an observation: Right now, lots of tools which depend on types (TypeScript Eslint, Typia, etc) run their own Lexer, parser and type checker. It is not uncommon to have separate type checkers running in the ESLint process, in a development server and in the TypeScript language service. This leads to doing a lot of work twice or more and can also lead to inconsistencies if the tooling does some weird stuff.

With an IPC layer it could become very feasible to have all of these hook into one tsgo process and have one source of truth for type information, meaning that the development experience when using these kinds of tools could get a lot smoother. Maybe this is something that can be considered when designing the IPC layer - i.e. to allow having one tsgo "server" and multiple clients consuming it's API.

[thebanjomatic]

Is now maybe the time to also re-consider adopting semver from the API perspective?

I could usually care less about breaking changes from the language and type checking perspective, but I would love it if from the tooling perspective we could get some sort of guarantees of what versions of the typescript api are supported and compatible.

I will acknowledge that in the past this was a harder sell because of the nature of javascript and how the entire AST and compiler internals are essentially all public api. As a result, any change or internal refactoring could therefore be considered a breaking change, and thus nothing was. If the team is being more intentional about what is exposed across the API boundary, it can be more reasonable.

My ideal would be something like:
major: release contains breaking changes to the api / lsp
minor: release contains new language features, some of which may introduce new errors
patch: mainly bug fixes

Note: changes to the language that add new features may change the resulting AST that is returned from the API. This should not be considered a breaking change to the API, but maybe things like removing documented properties from existing nodes, might be?

That way, tooling providers can set their expectations of supported typescript releases based on the ^major to state which version of the API their code is expecting to hit, and applications that are concerned about not getting new build errors can just use ~major.minor to pin things at the language level.

[RyanCavanaugh]

Absolutely; this is going to be mandatory going forward since it won't be apparent to API consumers what version they're talking to. A better versioning story is one of the reasons we want to make a more intentional design here. That may or may not correspond to the "language version", but we're going to have some concept of an "API version" that probably revs on a different axis.

[Brooooooklyn]

Hey there, I'm developing an experimental Node.js binding here: https://github.com/Brooooooklyn/typescript-go-napi/tree/main/rust

The basic idea is to compile typescript-go into a static link library, and then wrap it into a JavaScript API in Rust using NAPI-RS.

I have now completed two basic APIs, among which transform is like this:

import { transform } from '@typescript-go/api'

transform('const a: number = 1;', '/absolute/path/a.ts')

// 'const a = 1;\n'

[jakebailey]

Do note #455 (reply in thread)

This works but is likely not going to be safe for a number of important users.

[jakebailey]

I do actually have the code to do NAPI directly with Go, it'd be awesome if we could use it 😞

[Brooooooklyn]

This works but is likely not going to be safe for a number of important users.

Perhaps my implementation can be provided to users who don't need to load multiple different ts-go instances 👀

[jedwards1211]

has built a Node native module to use synchronous communication over standard I/O between external processes

How future-proof is this module? node-fibers used to be the defacto way of doing blocking calls, but it was eventually abandoned after significant V8 changes. I saw that coming years before it happened, everything in the ecosystem has gone toward async architecture.

Webdriverio badly wanted to provide a sync API for writing tests, and relied on node-fibers for this, but they eventually had to give up.

[jedwards1211]

@andrewbranch @jakebailey thoughts on how futureproof libsyncrpc is?

I guess the fact that it's based upon NAPI is better than node-fibers, but it still seems iffy

[jakebailey]

I don't really understand the question. This surely isn't worse than anyone else using napi-rs, and we're only using it to shuffle bytes around without hitting the event loop, not even constructing any objects or anything. It's possible that this could be implemented without NAPI by using plain reads/writes to file descriptors in JS, though that hasn't been tested in the newer IPC implementation.

[jedwards1211]

Okay yeah I hadn't realized that if native code simply makes blocking calls that will block the source JS function call (and event loop). node-fibers had to do something complicated to enable function calls to block local function execution while allowing the event loop and other functions to continue executing, and that magic apparently depended on pretty specific V8 APIs. I mistakenly assumed that anything else that would make a JS function call block would be similar

[HerringtonDarkholme]

Adding my two cents here, if API is not possible for major languages (mainly JS and probably Rust for SWC/OXC), a static type info snapshot can also be helpful. (though it is not possible to do transformation)

Related issues #504

[joshring]

Would it be possible to use tsc-go as a library from Go and hook into it that way? As opposed to plugins and IPC etc?

[jakebailey]

#481

[pjeby]

The TypeDoc project uses TypeScript compiler APIs (including watch-related APIs) to generate documentation from source projects (via JSDoc as well as TypeScript type info) and to respond to changes in the code base and non-code documentation files. Will the same or substantially similar watch APIs be available from JS? (i.e. including the ability to add arbitrary watches as well as incremental-compile-on-change)

[NullVoxPopuli]

How would we use something typedoc with ts plugins? like, supporting .vue, .svelte, .gjs, or .gts files?

[andrewbranch]

I think it’s very unlikely that any watch APIs will exist.

[andrewbranch]

The beginnings of the IPC API are now in a PR at #711.

[jakebailey]

No, you cannot make use of any of this right now. The APIs are going to be different, so that API example from the old repo is unlikely to work. What is in the current API package is a prototype, a proof of concept.

We're certainly aware of the annoyances in the old API, and want to make it better, but we have a challenge ahead of us as to how plugins are supposed to work since we can't actually load random code into a Go binary like we did a JS process, nor would there be any way to access anything internal (only what we explicitly expose).

[dummdidumm]

we can't actually load random code into a Go binary like we did a JS process

I can invoke Go from within JS using the bridge library you implemented (AFAIK that's what powers @typescript/api), are you saying there is no way to have a Go->JS call bridge?

[abhiaagarwal]

I was able to create a rust client for the IPC-based API in a few hours, the hardest part was the encoding format. Excited to see where this goes! https://github.com/abhiaagarwal/tsgo-rust-ipc/

[jakebailey]

I can invoke Go from within JS using the bridge library you implemented (AFAIK that's what powers @typescript/api), are you saying there is no way to have a Go->JS call bridge?

Sure, but only if we actually launch the server via that bridge.

[abhiaagarwal]

I don't get how the callbacks design doesn't already explicitly support this? Go is essentially invoking JS by asking it to return the results of a desired callback.

[agilgur5]

However the API is implemented, it would be great if the TS Compiler, Language Service, etc APIs were better documented and provided more detailed and complex examples than the currently limited wiki pages. Sample integrations and such could also be very useful.

Per microsoft/TypeScript#49790, microsoft/TypeScript#49050, and links therein, I've had to dive into the source code on numerous occasions to understand how to use pieces of the API and fix implementation issues, and sometimes haven't even been able to find an answer there either. This makes maintaining an integration really difficult in certain areas as there's little-to-no information to be found on proper usage, especially when it comes to various edge-cases like emitSkipped, type-only / emit-less files, and host OS filesystem nuances like symlinks / realpath, path normalization, etc, etc

[abhiaagarwal]

One limitation I've encountered as a (very early!) consumer of the API is that concurrency is not doable, in that due to the nature of pipes, I have to wait to consume a response from stdout before sending another request via stdin. I'd ideally want them to be able to operate independently. Is there any plan to address this? Could use an uds socket over stdin or maybe even correlation ids?

I was originally designing my API around async processing, but I'm switching to a mpsc approach with tsgo subprocess running in a dedicated thread. If the goal is for tsgo to become the canonical source of truth with multiple processes hooking into its API with it serving as more of a server, it'd be great to support a more traditional client-server architecture. UDS would be a good middlepoint between the overhead of tcp sockets

[andrewbranch]

Concurrent requests in a single session will definitely be supported eventually. I’m not sure whether “one server many clients” will be a generally supported pattern or not.

[abhiaagarwal]

One thing that would be nice (and I apologize, I don't mean to turn this into a feature request thread) off the top of my head is the ability to pass an AST directly to the api server. The big blocker for rust-based plugins like oxlint, biome, etc is the inconsistent mapping between their CST/AST and TS's AST (see typescript-eslint for the magic they do to convert back and forth). I could imagine an universe where a plugin creates their own internal AST, converts it to a TS AST (with an mapping from their nodes -> TS nodes), and then passes it directly to the API. Then, the linter can query the nodes they're interested in type information about directly without having to interpret typescript's AST back to theirs. The big blocker in the ecosystem from what I see is how ts interprets trivia in terms of leading spaces etc, which shouldn't matter for type-checking.

Basically, give tsgo the ability to deserialize an encoded SourceFile directly from an api client. This would also help stuff like tselint because clearly they know how to go from SourceFile -> their AST, doing the inverse saves tsgo from knowing what the files actually are (which Tsgo doesn't care about in the first place). There could also be something like a VirtualProject abstraction where tsgo can query for ASTs that the api client can return to them on-demand.

[Hookyns]

Well, I'm planning to create a similar system. I need something where, for example, an OXC or SWC transformer can query the TypeScript TypeChecker for type information about a symbol. I'll probably fork tsgo and build an API layer on top of it, unfortunately, since tsgo keeps everything internal, I have to create a custom build. But I want to keep it compatible so I can still merge changes from the original tsgo repository into my project.

It would be nice if tsgo exposed a proper Go API so I could use it as a package instead of having to fork it.

The motivation is to provide extended type inference capabilities for RTTIST - runtime type information system (reflection). Rewriting everything (RTTIST's typegen) in Go might make sense if tsgo became the standard for build tools, replacing SWC, esbuild, and OXC - but I don't think that's likely to happen.

[abhiaagarwal]

For what it's worth, I did look into this yesterday because I'm interested in doing the same thing (for oxc to solve oxc-project/oxc#3105) and I'm convinced it's possible: the API layer would have to expose the AcquireDocument and then, using the same binary format from the encoder (which the go implementation would need a decoder for), you can feed an ast.SourceFile into tsgo. The cool part is for ASTs you don't provide, tsgo will first query the DocumentStore, and if it can't find a match, it'll fallback to the VFS/FS and generate an AST for that file. So you could register the ASTs directly of files you're interested in and fallback for files whose ASTs you don't care about.

This is a cursory look and is probably incorrect in places, but the API is designed well-enough that this is feasible.

[Hookyns]

Ohhh… I have an OXC-based transformer where I need to infer types, and that’s why I’m doing this. I didn’t know about that issue, but I’m basically solving it. 😆

[tmm1]

Since --lsp --stdio is the only api currently available, I tried to wrap it as a standalone script using ts-lsp-client: https://github.com/tmm1/tsgo-linter-cli/blob/main/cli.ts

EDIT: got simple diagnostics working