TypeScript-powered Code Blocks

[orta]

Things I want for all code blocks inside website v2:

• Accurate Syntax Highlighting for complex TS (ex)
• Hover support for symbols (ex)
• Inline reporting of errors (ex)
• The ability to tell a code block to switch from TS code to any arbitrary part of the tsc emit (ex)
  • a JS file
  • a d.ts file
  • a source map
• The ability to create a working imports inside a codeblock (TS/JS/DTS)
• The ability to highlight lines, a line, or a range inside the line (ex)

Sidegoal:

Make this not directly tied to the website so that anyone with a Gatsby blog can re-use this for their own docs with very little setup.

Places with existing infra this can build from:

• orta/lsif-tsc-remark-example - an example of using the LSIF format (static language server protocol) to extract metadata. Overall, I think it's too much to go through the entire tsc-lsif, we could probably just steal the syntax for hover tokens and diagnostics and call it enough maybe.

• microsoft/TypeScript-New-Handbook the handbook has a good set of existing examples for this, these could be re-formatted into Gatsby/remark plugins.

• andrewbranch/gatsby-remark-vscode re-using VSCode at build time to handle the syntax highlighting.

• andrewbranch/blog - uses the typescript API to inline annotate embedded TypeScript ( long form on how it works )

/cc @andrewbranch @RyanCavanaugh

[orta]

Bikeshed: There are a bunch of ways in which embedded the complier metadata

@andrewbranch's JSON after language

'''ts { compiler: 'format' }
const myThing = ""
'''

and prefixed HTML comments (maybe with YML?)

<!--@
target: ES5
downlevelIteration: true
showEmit: true
-->
'''ts 
const str = "Hello!";
for (const s of str) {
    console.log(s);
}
'''
'''

@RyanCavanaugh's fourslash inspired inline comments:

'''
// @target: ES5
// @downlevelIteration
// @showEmit
const str = "Hello!";
for (const s of str) {
    console.log(s);
}
'''

My re-use of Jekyll-style which embed yml at the top of a code block:

'''
---
target: ES5
downlevelIteration: true
showEmit: true
---

const str = "Hello!";
for (const s of str) {
    console.log(s);
}
'''

I feel somewhat tied between fourslash-y or yml front-matter personally. After thinking about it a bit more, I'm in favor of fourslash-esk

[orta]

One thing that Andrew's blog does is that it allows for typescript files to to work across code samples, e.g.

<!--@
name: shapes.ts
-->
'''ts
interface Polygon {
  numberOfSides: number;
  sideLengths: number[];
}

enum TriangleKind {
  Accute = 'Accute',
  Right = 'Right',
  Obtuse = 'Obtuse'
}

interface Triangle extends Polygon {
  numberOfSides: 3;
  triangleKind: TriangleKind;
}

interface Quadrilateral extends Polygon {
  numberOfSides: 4;
  isRectangle: boolean;
}
'''

We have a base type `Polygon`, and two specializations that specify a number literal type for `numberOfSides`, along with some extra properties that are specific to polygons of their kind. This allows us to write a function that accepts either a `Triangle` or `Quadrilateral` and _discriminate_ between them based on the shape’s `numberOfSides`:

<!--@
name: shapes.ts
-->
'''ts
function addShape(shape: Triangle | Quadrilateral) {
  if (shape.numberOfSides === 3) {
    // In here, the compiler knows that `shape` is a `Triangle`,
    // so we can access triangle-specific properties.
    // See for yourself: hover each occurance of “shape” and
    // compare the typing info.
    console.log(shape.triangleKind);
  } else {
    // In here, the compiler knows that `shape` is a `Quadrilateral`.
    console.log(shape.isRectangle);
  }
}
'''

Where at build time shape.ts contains both the two code samples, this reduces example code duplication - but is it something this should support? For example the tsconfig page as it is would break today because you can't re-use filenames or examples with different compiler flags.

[robpalme]

In addition to passing compiler flags as comments, it would be great to also be able to control the tsc version.

This would help for tsc bug reporting and easing comms around tsc behaviour change over time.

[felixfbecker]

Something that might help with getting the hover tooltip: We have tsserver exposed over WebSockets at wss://typescript.sourcegraph.com. It speaks the Language Server Protocol, and can accept arbitrary textDocument/didOpen calls followed by textDocument/hover to get a Hover result. Without needing to clone a repo (but only sending over the snippet) I would expect this to be almost instant response.

For the UI part of this - the library that renders the hover tooltips on sourcegraph.com and in our browser extension is stand-alone and open source: https://github.com/sourcegraph/codeintellify (I always hoped one day it would be useful exactly for open source documentation snippets!)

[orta]

What a great idea @felixfbecker! if this were not for the TypeScript website primarily, I'd definitely be into that - we'll probably need a bunch of weird features like the above. ❤️

[andrewbranch]

A few random thoughts:

• I think we should specify that we’re not looking (at least I think we’re not looking) for editable code blocks, which would required a lot of heavy things running in the browser (or as external services like @felixfbecker) mentioned. Since these code blocks are intended to be static, all the analysis can be done at build time and written to JSON.
• I’m not super familiar with LSIF, but I was taking a look at examples in the specification, and it looks like hover responses come back as plain text. The TypeScript language service provides hover info as an array of classified text spans, which allows the hover tooltips to have accurate syntax highlighting in them. Is LSIF extensible/flexible enough to allow richer responses like that? I love the idea of using a standard format for this, but we would have to make sure it wouldn’t limit what we can display.
• I would look at this is as a design question in three separate parts:
  1. How do I consume input (raw code plus compiler options) and process it to get all the rich metadata I might want to display (syntax highlighting data, symbol info, diagnostic messages, highlight spans, etc.)?
  2. How do I structure all that into a static format?
  3. How do I render something rich and useful from that static format?

For (i), the answer is mostly the compiler API, but there are other providers you’ll need to get data from too—syntax highlighting isn’t going to come from the compiler (I tried it once; it doesn’t give good results), and line or span highlighting is going to be some kind of separate annotation. So for (ii), even if you want to use LSIF for part of this, you’ll probably need to embed the LSIF data within a top-level data structure that also holds syntax highlighting and other auxiliary data. Then (iii) is just a fun UI-building problem.

I should also mention that gatsby-remark-vscode is (appropriately) only designed to work within the Gatsby stack, and currently it only outputs HTML, essentially robbing you of the opportunity to do steps (ii) and (iii) yourself. I’m in the process of allowing users to query GraphQL for its tokenization results, which would let you compose its info with other data sources (ii) and render it all however you want (iii). But, if you’d rather work with something lower level or not gatsby-specific, but higher level than vscode-textmate, you could try shiki.

[RyanCavanaugh]

Something I'd also like is for a bot to show up on GitHub issues and show the typechecking/emit results of any code blocks in the OP, along with a short version history (e.g. "This behavior is unchanged since version 3.1")

[felixfbecker]

@orta do you mean specifying the compiler options? I think that can work like this:

• have the website JS read those options and construct a tsconfig.json JSON string
• call textDocument/didOpen with file:///tsconfig.json passing the content
• call textDocument/didOpen with file:///snippet.ts with the content of the snippet
• call textDocument/hover on hovers with the position to get a hover on

But if you can/want to precompute this data, that makes sense, and there would be no need to involve language servers then. codeintellify could still be of help here hopefully - it can handle the mouseover events and render the hover from a (statically embedded) LSP Hover result (you can completely control the rendering and styling if you want).

@andrewbranch since I've used LSIF and LSP a lot, maybe I can answer some of the questions: LSIF is probably a good bet (and is just JSON). Both LSIF and LSP use the Hover type, which usually contains a markdown string. That markdown string (and any code snippets inside) can then be rendered by any client markdown renderer and syntax highlighter, e.g. marked and highlight.js is what we use. That usually is good enough, as highlight.js is good at still handling these kind of "incomplete" code excerpts common in markdown.

I'm happy to answer any other questions or share my knowledge here if it's helpful - I think having hover tooltips in docs would be an amazing way to show how what makes TypeScript great!

[orta]

@felixfbecker - yeah, I want to pre-compute everything on the site, and it's not definitely not outside the realm of possibility that we'll want to run this sort of code against betas and PR builds also (like we do for the playground) - so having it all live in the build process is pretty much a must for me

@RyanCavanaugh @andrewbranch Yeah - thanks, I think this can be split into a multi-step process then:

• a library which takes a dumb fourslash-y string and returns the tsc emit, diagnostics and highlight metadata. This can be used both on the site, and in GitHub actions or bots etc. Then can be bootstrapped by the new handbook but with docs & tests etc. You can see this in Adds a new library for code samples #129

• a remark plugin that runs the library above when it finds'''ts which would be JS (or ts, or whatever)

• a separate remark plugin that takes TS/JS code and pulls out the spans in a way that we can then statically generate hover metadata for in some format

• the syntax highlighting plugin which should probably be last in the list of remarks and be responsible for generating the HTML

• the site can provide an example on how to make a JS front-end for showing the hover info

Each of these move it forwards

[orta]

There's been quite a lot of progress on this:

• ts-twoslasher is a lib which does the transformation and markup
• gatsby-remark-twoslasher-code-blocks fits in the markdown pipeline for gatsby and runs the code through
• gatsby-remark-shiki takes any codeblock and runs it through Shiki for syntax highlighting, but is also twoslash aware for things like highlights / errors etc

Example

[orta]

This is for all intents and purposes, done and the modules are shipped to npm