Editor parity: TypeScript-source debugging (M0-M2) and document symbols - #1308
Merged
Conversation
Adds `--debug`/`-g` to compile mode: the assembly is written with a debug directory and a matching portable PDB beside it. This is the symbol pipeline the TypeScript-source debugger is built on (epic #1306, track 0); sequence points arrive with the statement source model. Two properties of the toolchain made the obvious approach wrong. `PEPacker.AssemblyReferenceRewriter` rebuilds the final PE to retarget System.Private.CoreLib references, and the rebuilt image has no debug directory — it exposes only Rewrite/Save/Dispose, so there is no hook to preserve one. Symbols are therefore attached to the finished bytes: DebugDirectoryInjector appends the directory into a section's file-alignment slack, growing VirtualSize inside the section's own virtual page so no RVA, section count, or header size changes. The rewriter does preserve MethodDef row identity, which is what keeps a PDB valid across it, but PdbEmitter.VerifyMethodMappingPreserved proves that per build and fails the compile rather than shipping symbols that point a debugger at the wrong methods. The PDB is serialized against the finished image's row counts, since rewriting changes TypeRef/MemberRef/AssemblyRef counts. `PersistedAssemblyBuilder.GenerateMetadata` can populate a PDB metadata builder, but it adds a MethodDebugInformation row only for methods that have an IL body. That table is defined to be parallel to MethodDef, so every abstract or interface method shifts the rows after it — in a real build, 894 rows for 905 methods, and line information lands on the wrong method. SharpTS emits interfaces, so it always hits this. DebugInfoCollector builds the Document and MethodDebugInformation tables directly, keeping the parallel invariant exact and leaving room for hidden points and local scopes. SaveToBytes() keeps its contract for in-memory and test callers; SaveArtifacts() is the new PE-plus-PDB result, and Save(path, pdbPath) lets EXE builds put symbols beside the executable rather than the temporary DLL that gets bundled. Builds without --debug take none of this path and emit no debug directory. Tests cover the gate directly: an assembly shaped to expose the misalignment trap goes through the real rewriter, and the resulting PDB is checked for matching CodeView identity, checksum, documents, and sequence points landing on the methods they were recorded against.
Introduces the per-source model that TypeScript-source debugging and editor navigation both need: where every statement came from, carried intact across the rewrites that happen between parsing and emission (epic #1306, track A). SourceSpan holds half-open UTF-16 offsets — the form Token.Start already carries, and the one that nests and compares with plain integer arithmetic. Spans live in a SpanTable keyed by object reference rather than on the AST records: a span field would drag position into record equality, so two `return x;` statements on different lines would stop comparing equal. SourceDocument ties a file's text, checksum, line index, and span table together; ParsedModule now carries one, with the embedded stdlib marked virtual since it has no file a debugger could open. Span capture is centralized in Declaration() and Statement() rather than spread across the productions they dispatch to. Every declaration and statement in the language funnels through those two methods, so wrapping them covers all of them and cannot drift as productions are added. First-write-wins means the innermost production — which knows the tightest extent — is the one kept. Transforms have to state provenance explicitly, because they rebuild nodes: VarHoister rewrites every statement enclosing a `var`, and GeneratorArrowLifter rewrites everything around a lifted generator. Both now copy the original's position onto its replacement at the single point all rewrites pass through, and mark genuinely synthetic nodes hidden — the hoisted `var x;` declarations sit at the top of the body, not where the user wrote them, so pointing them at the original text would make stepping jump backwards. Destructuring lowerings attribute each part of the expansion to the declaration it came from. Token gains End and Span. The lexer builds every lexeme as the exact slice it consumed, so the end is the start plus the lexeme length, quotes and delimiters included. Splitting `>>`/`>>>` when closing nested generics now carries the offset forward instead of dropping it, which had been costing a position for everything parsed after a nested generic. Attributing a lowering's parts descends only into parts that lack a span. Without that guard the walk compounds — statement nodes are shared between sequences and a sequence is re-recorded by every enclosing production — which turned a two-file project check into a 19-second run. AttributingNestedLowerings StaysLinear guards it directly, since the CLI integration tests only caught it under parallel load and would have missed it on a faster machine. Expressions, type nodes, and the compilation-side NestedFunctionLifter are not instrumented yet; the lifter needs ILCompiler to receive documents, which arrives with sequence-point emission.
Emits sequence points for executable statements, so `--compile app.ts -g` produces an assembly a debugger steps through in the original `.ts` source rather than in emitted IL (epic #1306, track B1). Statement dispatch was duplicated: ILEmitter overrode EmitStatement with its own switch, and the state-machine emitters used the base one, so there was no single place to mark a statement. Both now override EmitStatementCore and share one EmitStatement, which is where the mark happens — after the dead-code check and before any IL, so the offset is the first instruction the statement produces. Async and generator bodies get line information from that alone, without either emitter knowing symbols exist. Two paths still needed attention, both found by tracing what actually reached the dispatcher rather than assuming. Top-level expression statements go through EmitExpressionWithAsyncWait, which wraps them in top-level-await plumbing and drives expression emission itself; it marks explicitly via MarkStatementStart. And containers were taking the offset of the statement inside them: a block or a `try` emits nothing of its own, so it shared an offset with its first real statement, and since two points cannot share an offset the container won — stepping landed on a brace. Blocks, lowering sequences, `try`, and labels no longer take a point; conditions and loop headers still do, because their test really does execute there. Type-only and separately-emitted declarations are skipped as well. Which document a statement belongs to is resolved from the module path emission already tracks, rather than a parallel "current document" every phase would have to maintain. CompilationContext gains the owning method and the debug scope, supplied by the context factories — the one place contexts are built. Debug builds are marked [Debuggable(Default | DisableOptimizations)] so the JIT keeps the shape the PDB describes. Builds without --debug set neither the scope nor the attribute and emit no points. Also completes the transform provenance deferred from the span work: NestedFunctionLifter copies positions onto the statements it rebuilds and marks the aliases it leaves behind hidden, which is possible now that documents reach ILCompiler. Verified by hand across multi-module, async, generator, loop, try/catch, class and hoisted-var programs — including that an imported module's methods resolve to their own file. docs/debugging-typescript.md carries the editor recipes and the manual smoke checklist; automated coverage asserts the line sets directly.
Emits the LocalVariable and LocalScope tables, so a debugger stopped in compiled TypeScript shows `doubled` rather than a slot number, and offers a binding only where it is actually in scope (epic #1306, track B2). LocalsManager is the seam. Every binding the user wrote is declared through it, while spill and scratch slots are declared straight on the ILGenerator and never reach it — so listening in that one place names user-visible locals and nothing else, rather than emitting everything and classifying afterwards. It reports declarations and scope boundaries through ILocalSymbolSink, which keeps it from depending on the symbol types; the context factories attach the sink, being the one place emission contexts are built. Scopes nest, so MethodLocalSymbols tracks them with a stack: entering records the IL offset a scope opens at, exiting records where it closes. The scope LocalsManager starts with is the method body itself and is never explicitly closed, so its length comes from the finished method's IL size, read back from the emitted image. Resolving a name by walking outward from the innermost scope is what makes shadowing work, and what keeps a `for` binding out of the locals window outside its loop. The format requires LocalScope to be sorted by method, then start ascending, then length descending, with each scope's variables in a contiguous run of LocalVariable rows; both are established when the tables are written, and a test pins the ordering because a reader silently resolves against the wrong enclosing scope if it drifts. Temporaries the lowerings introduce are marked DebuggerHidden. Two categories deliberately do not appear as locals, and the docs say so rather than leaving them as surprises: module top-level bindings are static fields of $Program because they outlive the initializer that assigns them, and async and generator locals are state-machine fields because they must survive suspension. Presenting the latter as locals again needs the portable-PDB state-machine hoisted-variable custom debug information, which is a follow-up.
One command to go from an open `.ts` file to a debug session: save if dirty, compile that saved source with `-g`, and start debugging the result (epic #1306, track B4). Saving first matters. The PDB records a SHA-256 checksum of the text that was compiled, so debugging unsaved-then-stale source is exactly the case a debugger refuses to bind breakpoints for; compiling what is on disk keeps the checksum describing what the editor is showing. No SharpTS debug adapter is involved. A compiled program is an ordinary .NET assembly, so the `coreclr` adapter from the C# extension drives it. That extension is a separate install and `startDebugging` fails unhelpfully without it, so the command checks for it up front and offers to install it. The docs also carry the `launch.json` form for driving it manually, since a launch configuration cannot be contributed for a debug type another extension owns. Output goes beside the source rather than into a temporary directory: that is where the runtimeconfig, co-located dependencies, and imported modules resolve from, and it leaves nothing to accumulate.
Marks methods compiled from embedded stdlib modules [DebuggerNonUserCode], so stepping over `path.join(...)` lands on the next line the user wrote instead of descending into `path.ts` (epic #1306, track B3). Emitting sequence points made this necessary. A Node-compatible module is real TypeScript compiled into the program, so once every statement carried line information the stdlib became exactly as steppable as the user's own files — a three-line program importing `path` gained a 43 KB PDB and a stack of stdlib frames to step through. Line information for those methods is deliberately kept rather than suppressed: a stack trace through the stdlib should still name a file and line, and turning Just My Code off should still let you step in. Only the classification changes. Emitted runtime helpers need no marking. They carry no debug information at all, which already places them outside user code — adding attributes there would be belt-and-braces on a rule the debugger already applies.
Adds a feature-mode contract to the language server and the first navigation capability behind it: `textDocument/documentSymbol` (epic #1306, track C1 and the first step of C3). The two audiences want opposite things. In VS Code these files are already served by tsserver, which does ordinary navigation better than SharpTS could and would collide with it; what SharpTS knows that tsserver does not is the .NET interop surface. A standalone client — Neovim, Helix, anything without a TypeScript server of its own — has no such alternative. So `sharpts-lsp` takes `--language-features interop-only|full` and defaults to `full`, while the VS Code extension launches it explicitly as `interop-only`. Interop diagnostics, hover, completion and signature help are served in both. The mode is fixed when the server starts, and deliberately so: registering a handler is what advertises its capability during initialization, and retracting one afterwards needs dynamic registration/unregistration, which this server does not implement. Reading a setting later could not take back a capability the client has already been told about, so changing the mode means a restart. Outline ranges come from the parser's span table, so a symbol covers exactly the text its declaration occupies and selecting one puts the cursor on the name rather than the whole declaration. Building it surfaced two declarations that never passed through the statement dispatcher and so carried no span at all: `export class C {}` parses its declaration directly, and namespace bodies have their own member dispatcher. Both are fixed where the spans are recorded rather than worked around in the outline, since every future consumer of positions would have hit the same hole. A declaration with no recorded span is skipped rather than given an invented range, and an unparseable file yields no symbols instead of throwing — a client asks for symbols on every keystroke, so half-typed input is the normal case.
The pieces were each covered, but nothing asserted them together on a program with imports, which is the shape the criterion is actually about: two TypeScript documents with matching checksums, sequence points attributed to the file each method was written in, named locals, nested lexical scopes, and a CodeView identity that still matches after the reference rewriter has rebuilt the assembly. Multi-document attribution had only been checked by hand until now — the existing assertion was Assert.Single on a one-document fixture, which cannot catch a method resolving to the wrong file. Also records in the docs why the remaining verification gap stays manual. No automated test observes a debugger actually stopping; `vsdbg` cannot be scripted for it, being licensed for use only with Visual Studio and VS Code and enforcing that with a handshake its own clients answer. Automating it means `netcoredbg`, which is a separate install, so the smoke checklist stays manual for now — and says so, rather than leaving the next person to rediscover it.
Both sides extended the same parser fluent chain in ModuleResolver.LoadModule: this branch added .WithSourceDocument(document) so each module keeps the text, checksum and statement spans that debug symbols and editor navigation resolve against; main added .WithFilePath(absolutePath) and the JSX dialect setup. All of them are kept — they configure different things and none supersedes another. WithSourceDocument only defaults the file path when none was set, so main's explicit WithFilePath still wins regardless of order.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Delivers the debugger half of #1306.
sharpts --compile app.ts -gnow produces an assembly adebugger steps through in the original
.tssource: breakpoints bind, stepping follows executablestatements, locals show under their source names within their real scopes, and Just My Code steps
over the bundled stdlib.
Milestones M0, M1 and M2 complete; M3 started. Full suite green after every commit
(15881 tests). #1306 stays open — see what's left, below.
Commits
3a14dc6f2c5b226b5f535503720a7bf2e902b137e3e4ad577b0577cc0d2536ccTwo findings that shaped the design
The go/no-go gate was passable, but not the obvious way.
PEPacker.AssemblyReferenceRewriterdestroys the debug directory and exposes no hook — but it preserves
MethodDefrow identity,which is what makes re-attaching symbols to the finished bytes valid.
DebugDirectoryInjectorappends the directory into
.text's file-alignment slack, changing no RVA, section count or headersize;
PdbEmitter.VerifyMethodMappingPreservedproves the assumption per build and fails thecompile rather than shipping symbols that point a debugger at the wrong methods.
A .NET 10 sharp edge.
PersistedAssemblyBuilder.GenerateMetadata(out pdbBuilder)adds aMethodDebugInformationrow only for methods that have an IL body, so the table is not parallelto
MethodDefas the format requires. SharpTS emits interfaces, so it always hits this — 894 rowsfor 905 methods, and every sequence point lands on the wrong method. SharpTS therefore builds the
PDB tables itself.
A performance regression, found and fixed
Attributing lowered statements re-walked shared subtrees, turning a two-file
sharpts -p .from698 ms into 19,286 ms. Three CLI integration tests caught it, but they passed in isolation and
failed only under suite load, so
AttributingNestedLoweringsStaysLinearguards it directly. The fixalso cut the whole suite from ~20 min to ~14.5 min.
Still open on #1306
(definition, references, rename). Deliberately not started: the epic prefers integrating binding
capture with checker/environment resolution over a divergent scope implementation, and
TypeSystem/is ~35.5k lines across 43TypeChecker*.csfiles. A partly-correct binder yieldswrong "go to definition", which is worse than the feature being absent.
Known verification gap
Symbol metadata is asserted thoroughly. No automated test observes a debugger actually stopping.
vsdbgcannot be scripted for it — it is licensed for use only with Visual Studio and VS Code andenforces that with a handshake its own clients answer — so automating this means
netcoredbg, aseparate install. Until then
docs/debugging-typescript.mdcarries a manual smoke checklist, whichhas not yet been run by a human. The VS Code command typechecks but was likewise not exercised
end-to-end.
Accepted v1 behaviour, documented rather than left as surprises: module top-level bindings appear as
static fields of
$Program(they outlive the initializer), and async/generator locals appear asstate-machine fields (they must survive suspension) — restoring the latter as locals needs
portable-PDB state-machine custom debug information.
Refs #1306