shutdown sent between initialize and initialized

[PerMildner]

Sublime Text (sometimes?) sends a shutdown request after initialize but before subsequent initialized and didOpen, see attached log. This is not a valid sequence of messages.

After shutdown the server should only accept exit so it is effectively stopped. The specification says "If a server receives requests after a shutdown request those requests should be errored with InvalidRequest".

Please include in your report:

• OS and language server
  macOS 10.14.6. Non-public language server
• How you installed LSP (Package Control or from git?)
  Installed with Package Control
• Minimal reproduction steps
  No open files or folders. Open a file associated to a language server. Not clear whether it happens every time.
• Log

startup, version: 3207 osx x64 channel: stable
executable: /Applications/Sublime Text.app/Contents/MacOS/Sublime Text
working dir: /
packages path: /Users/perm/Library/Application Support/Sublime Text 3/Packages
state path: /Users/perm/Library/Application Support/Sublime Text 3/Local
zip path: /Applications/Sublime Text.app/Contents/MacOS/Packages
zip path: /Users/perm/Library/Application Support/Sublime Text 3/Installed Packages
ignored_packages: ["Vintage"]
pre session restore time: 0.181761
using gpu buffer for window
startup time: 0.258246
environment variables loaded using: /bin/bash -l
reloading plugin Default.arithmetic
reloading plugin Default.auto_indent_tag
reloading plugin Default.block
reloading plugin Default.colors
reloading plugin Default.comment
reloading plugin Default.convert_color_scheme
reloading plugin Default.convert_syntax
reloading plugin Default.copy_path
reloading plugin Default.detect_indentation
reloading plugin Default.echo
reloading plugin Default.exec
reloading plugin Default.fold
reloading plugin Default.font
reloading plugin Default.goto_line
reloading plugin Default.history_list
reloading plugin Default.indentation
reloading plugin Default.install_package_control
reloading plugin Default.kill_ring
reloading plugin Default.mark
reloading plugin Default.new_templates
reloading plugin Default.open_context_url
reloading plugin Default.open_in_browser
reloading plugin Default.pane
reloading plugin Default.paragraph
reloading plugin Default.paste_from_history
reloading plugin Default.profile
reloading plugin Default.quick_panel
reloading plugin Default.rename
reloading plugin Default.run_syntax_tests
reloading plugin Default.save_on_focus_lost
reloading plugin Default.scroll
reloading plugin Default.set_unsaved_view_name
reloading plugin Default.settings
reloading plugin Default.show_scope_name
reloading plugin Default.side_bar
reloading plugin Default.sort
reloading plugin Default.switch_file
reloading plugin Default.symbol
reloading plugin Default.transform
reloading plugin Default.transpose
reloading plugin Default.ui
reloading plugin CSS.css_completions
reloading plugin Diff.diff
reloading plugin HTML.encode_html_entities
reloading plugin HTML.html_completions
reloading plugin ShellScript.ShellScript
reloading plugin 0_package_control_loader.00-package_control
reloading plugin 0_package_control_loader.01-pygments
reloading plugin 0_package_control_loader.50-markupsafe
reloading plugin 0_package_control_loader.50-pymdownx
reloading plugin 0_package_control_loader.50-python-markdown
reloading plugin 0_package_control_loader.50-pyyaml
reloading plugin 0_package_control_loader.51-python-jinja2
reloading plugin 0_package_control_loader.55-mdpopups
reloading plugin LSP.boot
reloading plugin Package Control.1_reloader
reloading plugin Package Control.2_bootstrap
reloading plugin Package Control.Package Control
plugins loaded
LSP: global configs ['jdtls=False', 'phpls=False', 'flow=False', 'reason=False', 'cquery=False', 'clangd=False', 'lsp-tsserver=False', 'eslint=False', 'ocaml=False', 'ruby=False', 'javascript-typescript-langserver=False', 'pyls=False', 'haskell-ide-engine=False', 'ra-lsp=False', 'polymer-ide=False', 'gopls=False', 'vscode-css=False', 'intelephense-ls=False', 'metals=False', 'typescript-language-server=False', 'golsp=False', 'dart=False', 'spider-nc=True', 'rls=False', 'bashls=False']
LSP: Couldn't determine project directory since no folders are open and the current file isn't saved on the disk.
LSP: window 2 starting 1 initial views
LSP: window 2 requests spider-nc for /Users/perm/sicstus/lsptestfolder/foo.pl
LSP: Couldn't determine project directory since no folders are open! Using /Users/perm/sicstus/lsptestfolder as a fallback.
LSP: starting in /Users/perm/sicstus/lsptestfolder
LSP: starting ['/usr/bin/nc', 'localhost', '56789']
LSP:  --> initialize
LSP: window 2 added session spider-nc
LSP: Couldn't determine project directory since no folders are open! Using /Users/perm/sicstus/lsptestfolder as a fallback.
LSP: project path changed, ending existing sessions
LSP: new path = /Users/perm/sicstus/lsptestfolder
LSP: unloading session spider-nc
LSP:  --> shutdown
Package Control: Skipping automatic upgrade, last run at 2019-08-01 16:21:54, next run at 2019-08-01 17:21:54 or after
LSP:      {'capabilities': {'documentFormattingProvider': True, 'implementationProvider': False, 'codeActionProvider': False, 'callHierarchyProvider': False, 'renameProvider': True, 'typeDefinitionProvider': False, 'workspaceSymbolProvider': True, 'definitionProvider': True, 'documentRangeFormattingProvider': True, 'workspace': {'workspaceFolders': {'supported': True, 'changeNotifications': True}}, 'typeHierarchyProvider': False, 'documentHighlightProvider': True, 'referencesProvider': True, 'textDocumentSync': 2, 'hoverProvider': True, 'documentSymbolProvider': True, 'completionProvider': {'triggerCharacters': ['(']}}}
LSP:  --> initialized
LSP:  --> textDocument/didOpen
LSP:      None
LSP:  --> exit
LSP: session spider-nc ended
LSP: clients for window 2 unloaded
LSP: window 2 requests spider-nc for /Users/perm/sicstus/lsptestfolder/foo.pl
LSP: Couldn't determine project directory since no folders are open! Using /Users/perm/sicstus/lsptestfolder as a fallback.
LSP: starting in /Users/perm/sicstus/lsptestfolder
LSP: starting ['/usr/bin/nc', 'localhost', '56789']
LSP:  --> initialize
LSP: window 2 added session spider-nc
LSP: LSP stdout process ended.
using gpu buffer for window
Unable to open /Users/perm/Library/Application Support/Sublime Text 3/Packages/Default/Preferences.sublime-settings
LSP: Couldn't determine project directory since no folders are open! Using /Users/perm/Library/Application Support/Sublime Text 3/Packages/User as a fallback.
LSP: window 3 requests spider-nc for /Users/perm/Library/Application Support/Sublime Text 3/Packages/Default/Preferences.sublime-settings
LSP: Couldn't determine project directory since no folders are open! Using /Users/perm/Library/Application Support/Sublime Text 3/Packages/User as a fallback.
LSP: starting in /Users/perm/Library/Application Support/Sublime Text 3/Packages/User
LSP: starting ['/usr/bin/nc', 'localhost', '56789']
LSP:  --> initialize
LSP: window 3 added session spider-nc
LSP: LSP stdout process ended.
LSP: no config found or enabled for view /Users/perm/Library/Application Support/Sublime Text 3/Packages/Default/Preferences.sublime-settings
LSP: no config found or enabled for view /Users/perm/Library/Application Support/Sublime Text 3/Packages/Default/Preferences.sublime-settings
LSP: Couldn't determine project directory since no folders are open! Using /Users/perm/Library/Application Support/Sublime Text 3/Packages/User as a fallback.
LSP: Couldn't determine project directory since no folders are open! Using /Users/perm/Library/Application Support/Sublime Text 3/Packages/User as a fallback.
LSP: Couldn't determine project directory since no folders are open and the current file isn't saved on the disk.
LSP: project path changed, ending existing sessions
LSP: new path = None
LSP: unloading session spider-nc
LSP:  --> shutdown
Failure writing to stdout
Traceback (most recent call last):
  File "/Users/perm/Library/Application Support/Sublime Text 3/Installed Packages/LSP.sublime-package/plugin/core/transports.py", line 227, in write_stdin
    self.process.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe

LSP: Communication to server closed, exiting
LSP: transport failed
LSP: Couldn't determine project directory since no folders are open! Using /Users/perm/sicstus/lsptestfolder as a fallback.
reloading settings Packages/User/Preferences.sublime-settings
reloading settings Packages/User/Preferences.sublime-settings
reloading settings Packages/User/Preferences.sublime-settings
reloading settings Packages/User/Preferences.sublime-settings

[tomv564]

The shutdown message is sent because LSP thinks you have opened a different project.

LSP is also logging that you have not created a sublime project for your 'lsptestproject' folder.
Create a project for this folder and I bet this issue will disappear.
LSP tries to handle support project-less operation on a best-effort basis, but bugs like these are expected.

Explanation: LSP detects a supported file (foo.pl) for which it starts a session in lsptestfolder and then an unsaved file is opened for which project_path is None. When project path changes from we assume you've stopped working in lsptestfolder, so we end that session.

This logic works reliably with sublime projects but needs substantial changes for project-less windows. If someone can find a really elegant way to manage sessions then please propose a design in a new issue or PR.

[tomv564]

I feel LSP does too little to warn users about project-less mode. Perhaps we can either:

• Disable it behind a setting flag (this will cause many more "it's not working" issues)
• Add a warning dialog every time a window starts a project-less session.

[PerMildner]

The problem is not so much the shutdown message (and the missing exit message that ought to follow the shutdown).
The problem is that didOpen is sent to a server that Sublime Text knows it has shut down. This is a LSP protocol violation.

If a proper shutdown (shutdown + exit) was followed by a proper startup (launch server, initialize, ...) before the didOpen, then it would perhaps not work perfectly for project-less files, but at least it would adhere to the LSP protocol. As it now stands a conformant server will just report a InvalidRequest error for all subsequent requests.

Perhaps it would be possibly to simply start one server instance for each project-less file? Or, not use the single server instance except for the first file, and not use LSP for subsequent files.

Other clients seems to just default to the surrounding folder as rootURI (and workspace folder) so everything works OK as long as all files are in, or below, that initial folder. Then, there is the workspace/didChangeWorkspaceFolders LSP-message that perhaps could be used to just add any new containing folder. I know nothing about ST projects, though.

[PerMildner]

Looking some more at this. It seems exit is sent, at least sometimes, some time after the shutdown, but I do not see it always (so, there is not a fundamental bug that "exit is not sent after shutdown").

Looking at the source (*) I suspect a thread synchronization issue. I see nothing that prevents the didOpen (which seems to be posted from its own async thread) from getting sent to the server after a shutdown has been sent.

Also, once, when flipping around among the file tabs in Sublime Text, which triggers the shutdown/restart/whatever of the server, it appeared that several shutdown messages was buffered somewhere and then all was received by the same server. Since there are so many moving parts, and timing is involved, this is really hard to troubleshoot, so this particular symptom may be a false alarm.

(*) Be aware that I know nothing about the Sublime Text API, nor of Python, so this is just informed guesswork.

[tomv564]

Yeah, it's hard to reason about it - improving behaviour in single-file mode is the first priority as it can be hiding other bugs as well.
There is no "shutting down" state for the session, but that would prevent a late didOpen being sent to it.
Let's keep this issue open to fix the single-file mode first, then look at the behaviour again.

[tomv564]

I think with #713 some of the unintended shutdowns should have disappeared.
From your logs, I am not sure why your spider-nc config is being started for a preferences edit:

window 3 requests spider-nc for /Users/perm/Library/Application Support/Sublime Text 3/Packages/Default/Preferences.sublime-settings

I'll close the issue now without adding a "shutting down" state to sessions - assuming LSP will rarely have a legitimate reason to shutdown sessions that were just started.