Segmentation fault / nil pointer in `textDocument/didOpen`

[richardkmichael]

Stack trace

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x10139aec0]

goroutine 36 [running]:
github.com/microsoft/typescript-go/internal/project.(*Session).cancelDiagnosticsRefresh(0x14000221060?)
        github.com/microsoft/typescript-go/internal/project/session.go:309 +0x20
github.com/microsoft/typescript-go/internal/project.(*Session).DidOpenFile(0x0, {0x10180d1d8, 0x14000255800}, {0x140001602d0, 0x43}, 0x1, {0x14000162540, 0x5a}, {0x14000111db0, 0xa})
        github.com/microsoft/typescript-go/internal/project/session.go:180 +0x4c
github.com/microsoft/typescript-go/internal/lsp.(*Server).handleDidOpen(0x14000200b40?, {0x10180d1d8?, 0x14000255800?}, 0x14000221060?)
        github.com/microsoft/typescript-go/internal/lsp/server.go:676 +0x3c
github.com/microsoft/typescript-go/internal/lsp.init.func1.registerNotificationHandler[...].5({0x10180d1d8?, 0x14000255800}, 0x14)
        github.com/microsoft/typescript-go/internal/lsp/server.go:476 +0x54
github.com/microsoft/typescript-go/internal/lsp.(*Server).handleRequestOrNotification(0x14000000140, {0x10180d1d8, 0x14000255800}, 0x140001d28d0)
        github.com/microsoft/typescript-go/internal/lsp/server.go:424 +0xf4
github.com/microsoft/typescript-go/internal/lsp.(*Server).dispatchLoop.func1(...)
        github.com/microsoft/typescript-go/internal/lsp/server.go:329
github.com/microsoft/typescript-go/internal/lsp.(*Server).dispatchLoop(0x14000000140, {0x10180d210?, 0x1400015aa00?})
        github.com/microsoft/typescript-go/internal/lsp/server.go:347 +0x588
github.com/microsoft/typescript-go/internal/lsp.(*Server).Run.func1()
        github.com/microsoft/typescript-go/internal/lsp/server.go:225 +0x24
golang.org/x/sync/errgroup.(*Group).Go.func1()
        golang.org/x/sync@v0.17.0/errgroup/errgroup.go:93 +0x4c
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
        golang.org/x/sync@v0.17.0/errgroup/errgroup.go:78 +0x90

Remarks

I'm learning LSP with a simple client, so this crash could be due to misuse of the protocol.

The crash is textDocument/didOpen, during which there seems to be a nil Session pointer. I didn't see anything session related in the LSP specification, so I'm unsure if there's missing initialization in the simple "client" I implemented.

However, I also tested against the typescript-language-server project, which I believe wraps the classic tsserver to provide a few LSP commands. It does not segfault and responds correctly.

So I don't think tsgo should segfault. If it's a protocol misuse, an error message would helpful.

VSCode does not cause the crash in textDocument/didOpen. Unfortunately, the VSCode LSP debug output does not include the initialization messages. It would be helpful if VSCode logged the entire conversation, including initialization. I assume it is doing more on startup and establishing the Session struct. Debug output Included below.

Perhaps also interesting is that VSCode executes textDocument/diagnostic immediately, while the simple client does not, and the stacktrace is includes cancelDiagnosticsRefresh.

I tried adjusting the tsconfig.json, adding checkJs: false and noEmit: true, the segfault still occurred.

Steps to reproduce

1. Create tsconfig.json as:

{
  "compilerOptions": {
    "target": "ES2022",

    "module": "NodeNext",
    "moduleResolution": "NodeNext",

    "allowJs": true
  },
  "files": [
    "repro.js"
  ]
}

2. Create a repro.js as:

const getRandomValue = () => {
  return Math.random();
};

console.log(getRandomValue());

3. Create the Python (no dependencies) lsp_test.py "client" as:

Python `lsp_test.py`

The client was intended to experiment with workspace/symbol and textDocument/definition, but uses textDocument/didOpen.

import subprocess
import json
import os
import sys
import argparse
import time


def send_message(process, message):
    """Encodes and sends a JSON-RPC message."""
    try:
        json_message = json.dumps(message)
        encoded_message = json_message.encode("utf-8")
        content_length = len(encoded_message)

        full_message = f"Content-Length: {content_length}\r\n\r\n{json_message}"

        print(f"==> SENT:\n{full_message}\n", flush=True)

        process.stdin.write(full_message.encode("utf-8"))
        process.stdin.flush()
    except BrokenPipeError:
        # This can happen if the server crashes while we are trying to write
        print(
            "--- Could not send message: Broken pipe. Server is likely down. ---",
            file=sys.stderr,
        )


def read_message(process):
    """Reads and decodes a single JSON-RPC message from the server."""
    try:
        header = process.stdout.readline().decode("utf-8")
        if not header or not header.startswith("Content-Length"):
            return None  # Pipe was closed

        content_length = int(header.split(":")[1].strip())
        process.stdout.readline()  # Consume the blank line

        body = process.stdout.read(content_length).decode("utf-8")
        print(
            f"<== RECV:\nContent-Length: {content_length}\r\n\r\n{body}\n", flush=True
        )

        return json.loads(body)
    except (IOError, ValueError):
        return None


def main():
    parser = argparse.ArgumentParser(description="LSP Test Client.")
    parser.add_argument("server_cmd", type=str, help="Command to start the LSP server.")
    parser.add_argument(
        "--definition-direct",
        action="store_true",
        help="Test textDocument/definition directly.",
    )
    args = parser.parse_args()

    PROJECT_PATH = os.getcwd()
    PROJECT_URI = f"file://{PROJECT_PATH}"
    print(f"--- Using project path: {PROJECT_PATH} ---")

    print(f"--- Starting LSP server: {args.server_cmd} ---")
    process = subprocess.Popen(
        args.server_cmd.split(),
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=sys.stderr,
        bufsize=0,
    )

    request_id = 1
    try:
        # 1. Initialize
        initialize_request = {
            "jsonrpc": "2.0",
            "id": request_id,
            "method": "initialize",
            "params": {"processId": None, "rootUri": PROJECT_URI, "capabilities": {}},
        }
        send_message(process, initialize_request)

        response_found = False
        while not response_found:
            if process.poll() is not None:
                print(
                    "--- Server process terminated before initialize response. ---",
                    file=sys.stderr,
                )
                return
            response = read_message(process)
            if response and response.get("id") == request_id:
                response_found = True
        request_id += 1

        # 2. Open the file
        FILE_PATH = os.path.join(PROJECT_PATH, "repro.js")
        with open(FILE_PATH, "r") as f:
            file_content = f.read()
        did_open_notification = {
            "jsonrpc": "2.0",
            "method": "textDocument/didOpen",
            "params": {
                "textDocument": {
                    "uri": f"file://{FILE_PATH}",
                    "languageId": "javascript",
                    "version": 1,
                    "text": file_content,
                }
            },
        }
        send_message(process, did_open_notification)

        time.sleep(0.5)  # Give server a moment to process didOpen

        if process.poll() is not None:
            print("--- Server process terminated after didOpen. ---", file=sys.stderr)
            return

        if args.definition_direct:
            # --- MODE 2: Definition ---
            print("\n--- Running in --definition-direct mode ---")
            definition_request = {
                "jsonrpc": "2.0",
                "id": request_id,
                "method": "textDocument/definition",
                "params": {
                    "textDocument": {"uri": f"file://{FILE_PATH}"},
                    "position": {"line": 4, "character": 12},
                },
            }
            send_message(process, definition_request)
            read_message(process)
        else:
            # --- MODE 1: Workspace Symbol ---
            print("\n--- Running in default workspace/symbol mode ---")
            symbol_request = {
                "jsonrpc": "2.0",
                "id": request_id,
                "method": "workspace/symbol",
                "params": {"query": "getRandomValue"},
            }
            send_message(process, symbol_request)
            read_message(process)

    finally:
        print("--- Cleaning up process ---")
        # THE FIX: Only try to shut down the server if it's still running.
        if process.poll() is None:
            shutdown_req = {
                "jsonrpc": "2.0",
                "id": request_id + 1,
                "method": "shutdown",
                "params": None,
            }
            send_message(process, shutdown_req)
            # We don't need to wait for a response to shutdown
            exit_notif = {"jsonrpc": "2.0", "method": "exit", "params": None}
            send_message(process, exit_notif)

        process.terminate()
        print("--- Test finished ---")


if __name__ == "__main__":
    main()

4. Install tsgo and typescript-language-server (for comparison)

npm install @typescript/native-preview
npm install typescript-language-server typescript

5. Run the simple client with various Language Servers:

# Segfault
python lsp_test.py 'npx tsgo --lsp --stdin'

# No segfault
python lsp_test.py 'npx typescript-language-server --stdin'

Additional information

VSCode LSP debug output

[Trace - 1:06:51 PM] Sending notification 'textDocument/didOpen'.
Params: {
    "textDocument": {
        "uri": "file:///Users/rmichael/Documents/Personal/Source/lsp-repro/repro.js",
        "languageId": "javascript",
        "version": 1,
        "text": "const getRandomValue = () => {\n  return Math.random();\n};\n\nconsole.log(getRandomValue());\n"
    }
}


[Trace - 1:06:51 PM] Sending request 'textDocument/diagnostic - (17)'.
Params: {
    "textDocument": {
        "uri": "file:///Users/rmichael/Documents/Personal/Source/lsp-repro/repro.js"
    }
}


[Trace - 1:06:51 PM] Received response 'textDocument/diagnostic - (17)' in 7ms.
Result: {
    "kind": "full",
    "items": []
}


[Trace - 1:06:51 PM] Sending request 'textDocument/documentSymbol - (18)'.
Params: {
    "textDocument": {
        "uri": "file:///Users/rmichael/Documents/Personal/Source/lsp-repro/repro.js"
    }
}


[Trace - 1:06:51 PM] Received response 'textDocument/documentSymbol - (18)' in 4ms.
Result: [
    {
        "name": "getRandomValue",
        "kind": 13,
        "range": {
            "start": {
                "line": 0,
                "character": 6
            },
            "end": {
                "line": 2,
                "character": 1
            }
        },
        "selectionRange": {
            "start": {
                "line": 0,
                "character": 6
            },
            "end": {
                "line": 0,
                "character": 20
            }
        },
        "children": []
    }
]


[Trace - 1:06:52 PM] Sending request 'textDocument/documentSymbol - (19)'.
Params: {
    "textDocument": {
        "uri": "file:///Users/rmichael/Documents/Personal/Source/lsp-repro/repro.js"
    }
}


[Trace - 1:06:52 PM] Received response 'textDocument/documentSymbol - (19)' in 1ms.
Result: [
    {
        "name": "getRandomValue",
        "kind": 13,
        "range": {
            "start": {
                "line": 0,
                "character": 6
            },
            "end": {
                "line": 2,
                "character": 1
            }
        },
        "selectionRange": {
            "start": {
                "line": 0,
                "character": 6
            },
            "end": {
                "line": 0,
                "character": 20
            }
        },
        "children": []
    }
]


[Trace - 1:29:01 PM] Sending request 'textDocument/diagnostic - (20)'.
Params: {
    "textDocument": {
        "uri": "file:///Users/rmichael/Documents/Personal/Source/lsp-repro/repro.js"
    }
}


[Trace - 1:29:01 PM] Received response 'textDocument/diagnostic - (20)' in 20ms.
Result: {
    "kind": "full",
    "items": []
}

[jakebailey]

This can only happen if session is nil.

The session is created during the handling of the initialized notification post-handshake. I am not sure how that is possible, because we do treat initialized as a request that must be strictly ordered / handled synchronously and so no message that is read after initialized will ever be handled before initialized has finished setting things up.

Unfortunately, the VSCode LSP debug output does not include the initialization messages. It would be helpful if VSCode logged the entire conversation, including initialization. I assume it is doing more on startup and establishing the Session struct. Debug output Included below.

It definitely does, in the LSP output panel. Consider clearing it then restarting the server in case it was cut off or something.

Maybe my understanding of the LSP is off and the client is able to send a didOpen before it sends initialized, but I really don't think that's possible.

[richardkmichael]

This can only happen if session is nil.

The session is created during the handling of the initialized notification post-handshake. I am not sure how that is possible, because we do treat initialized as a request that must be strictly ordered / handled synchronously and so no message that is read after initialized will ever be handled before initialized has finished setting things up.

Unfortunately, the VSCode LSP debug output does not include the initialization messages. It would be helpful if VSCode logged the entire conversation, including initialization. I assume it is doing more on startup and establishing the Session struct. Debug output Included below.

It definitely does, in the LSP output panel. Consider clearing it then restarting the server in case it was cut off or something.

Thanks for mentioning this. (I had tried closing the file, clearing Output and re-opening the file, and didn't notice the server remained running.)

Maybe my understanding of the LSP is off and the client is able to send a didOpen before it sends initialized, but I really don't think that's possible.

Got it; my simple client isn't sending initialized, that fixed it. So this is protocol misuse. Sorry, I was mislead by the more tolerant typescript-language-server wrapper. I wonder if tsgo could send a "client not initialized" error message? Could be nice polish in the future.

Thank you very much for the fast reply!

[jakebailey]

I sent #1747. If you're extremely bored, it'd be helpful to try this PR with your client to see if it fails in a better way.

[richardkmichael]

Since it already passed through the merge queue, I built and tested main. It works well, thank you!

{"jsonrpc":"2.0","error":{"code":-32002,"message":"ServerNotInitialized"}}

Yesterday I had reviewed the spec, server behaviour prior to initialize is covered (but not initialized):

If the server receives a request or notification before the initialize request, it should act as follows:

• For a request, the response should be an error with code: -32002. The message can be picked by the server.

Which is slightly interesting, in that behaviour between them is unspecified. Seems reasonable to -32002, in just the same way.

To let the client know it is responsible, I might've made the message: ClientDidNotSendInitialized, or InitializedNotReceived (or whatever project terminology is precedent), because it's not obvious what server conditions result in "being initialized", i.e., processing project files? I would've wondered this while debugging, but even with ServerNotInitialized it probably would've been less time than I spent. 😂 (Sending me to lifecycle spec.)

Sorry I didn't get to this before it was merged, the project moves fast! 🏎️ Thanks again!