Increased time to first diagnostic/language server initialization with 1.14.2

[mjlbach]

Describe the bug
@clason and I noticed an increase in time to first diagnostic on macOS for 1.14.2. It used to provide diagnostics almost instantly on a file, but now it seems to take approximately 30 seconds before giving some timeout messages

LSP[sumneko_lua][Info] Too large file: test/unit/viml/expressions/parser_tests.lua skipped
. The currently set size limit is: 100 KB, and the file size is: 212.211 KB.
LSP[sumneko_lua][Info] Too large file: test/functional/plugin/shada_spec.lua skipped. The
currently set size limit is: 100 KB, and the file size is: 104.13 KB.
LSP[sumneko_lua][Info] Too large file: test/unit/eval/typval_spec.lua skipped. The current
ly set size limit is: 100 KB, and the file size is: 119.11 KB.

To Reproduce
Steps to reproduce the behavior:

1. Clone the neovim source repo
2. Open diagnostic.lua
3. runtime/lua/vim/lsp/diagnostic.lua
4. Wait 30 seconds

Expected behavior
Prior to 723f3d2 diagnostics were pretty much instant. This commit does reduce CPU usage though at idle, which is great.

Environment (please complete the following information):

• OS: macOS
• Client: neovim

[clason]

Same here (Intel macOS); last good version (no error, instant response on startup) was 1.12.0 (despite the hang in the test); going back to 1.13.0 already shows the error, but the timeout is much quicker (only a few seconds).

[sumneko]

I cannot reproduce this in my computer(Windows 10).
If it is caused by 723f3d2, it may be fixed when I done this: #329 (comment).

[clason]

No, I assume this is something specific to macOS (and possibly neovim); otherwise I'd have expected someone to have already reported this days ago ;)

So there are two regressions here:

1. the "Too large file" error, which didn't happen before 1.13;
2. the time it takes for the error, which was much shorter before 1.14.

The second issue may indeed be caused by that commit, but 1. seems to have a different cause. I'll try to bisect it in the next hours.

[sumneko]

1. the "Too large file" error, which didn't happen before 1.13;

This is not an error, but some large files were skipped for performance. You can control this size in the settings, the default is 100k. Perhaps in a certain version, your project has files that just exceed the threshold.

[clason]

Sorry, "warning" would have been better. Nevertheless, this did not occur previously, and I got instant diagnostics/completions. (Not complaining, mind you -- there's probably a good reason for it! Just trying to understand what is happening and how to get back the old behaviour.)

[sumneko]

Sorry, "warning" would have been better. Nevertheless, this did not occur previously, and I got instant diagnostics/completions. (Not complaining, mind you -- there's probably a good reason for it! Just trying to understand what is happening and how to get back the old behaviour.)

Oh, relax, I didn't feel any discomfort. My wording and tone may be a bit strange, it is all the fault of Google Translate. I'm just worried that you might have misunderstood this prompt, so I explained it a little bit.
By the way, I tried to open your project with version 1.12 on my computer, and I can see these tips.

[clason]

No problem, I just wanted to make sure -- as you write, it's always better to explain!

So I bisected and, not surprisingly, got 43a29d8 as the first bad commit.

I started from 1.12.2, which does give a tip, but it's a different one:

LSP[sumneko_lua][Info] Preloaded files has reached the upper limit (1000), you need to manually open the files
 that need to be loaded.

I also get completions before this hint appears. (The message changes over the next few commits, offering actions, as you know.)

[sumneko]

It seems that your project needs to get more lua scripts through other means. In my test, there were only more than 400 lua files in the work area, so it seemed that the initialization was completed quickly.

Due to the need to calculate global variables and require references, the correct approach is indeed to wait for the initialization to complete before responding to operations such as automatic completion. However, considering the following two reasons, in the previous version, I allowed to respond before the initialization was completed:

1. The cache will be refreshed every time a file is read or a file changes, so as your input, the auto-completed result will be more and more correct.
2. If there is no response from the automatic completion, the user will be confused.

But then the situation changed, so I changed to wait for the initialization to complete:

1. During continuous input, auto-complete will directly use the last result without recalculation, so the result is always incorrect as you input.
2. The progress protocol is supported, and the user can learn the current initialization status from the progress bar, and understand that it is reasonable that the automatic completion is not responding.

[clason]

Thank you for the explanation, that makes sense. So the wait is needed to guarantee correct results, and the only issue is that -- at least on macOS -- increasing the sleep value by 10x also increases the time it takes (from ~3s on 1.13 to ~30s in 1.14).

One thing that is still a bit confusing is that I also have to wait 30s for diagnostics if I open a new file test.lua in an empty directory (and no workspace defined in the settings). I don't get a timeout warning in this case, though.

[sumneko]

Please send me the log and see what I am doing in these 30 seconds.
See https://github.com/sumneko/lua-language-server/wiki/Default-log-path

In addition, please help test it, manually change 0.01 second back to 0.001 second result

[clason]

Sure! Here's the logs from an empty test file for an empty directory (it seems it's still indexing the lua-language-server directory, which explains why it's still taking time):
log.zip

[clason]

And, yes, decreasing the thread.sleep time down to 0.001 gives diagnostics after 3s instead of 30s. (But idle CPU load increases again from ~0.5% to ~3.3% -- still better than originally, I think? I notice that the thread.sleep in brave.lua is now gone completely.)

[sumneko]

Sure! Here's the logs from an empty test file for an empty directory (it seems it's still indexing the lua-language-server directory, which explains why it's still taking time):
log.zip

This log shows that initialization was completed in 0.4 seconds and only 12 meta files were loaded.

And, yes, decreasing the thread.sleep time down to 0.001 gives diagnostics after 3s instead of 30s. (But idle CPU load increases again from ~0.5% to ~3.3% -- still better than originally, I think? I notice that the thread.sleep in brave.lua is now gone completely.)

But it doesn't seem to matter anymore. The problem is that the behavior of sleep on the macOS platform is more complicated than I thought.

This problem is indeed caused by 723f3d2 (increasing the initialization time from 3 seconds to 30 seconds), and 43a29d8 exposed the problem (need to wait for the initialization to complete before responding).

I will temporarily change sleep back to 0.001 seconds, and then use #329 (comment) to completely solve the problem.

[clason]

I'm really sorry to say this, but since 1.15 I no longer get any diagnostics or completion (both on macOS and Linux). (CPU time is at 0% now, though...)

Log files:
log.zip

[sumneko]

Oh, I'm in holiday in next 7 days, have no machine to debug this.
You could only rollback for now.

[clason]

No worries, happy Lunar New Year! Version 1.14.3 works well (not too long delay, and not too much CPU use).

[sumneko]

My preliminary inspection found that this was caused by the inconsistent behavior of the os.clock function on the Windows platform and the Linux platform. I am still conducting further research.

EDIT: I found this: https://stackoverflow.com/questions/27178789/c-different-implementation-of-clock-in-windows-and-other-os. I will use other methods to achieve related functions.

[sumneko]

@clason I have fixed it, please check out and test it, thank you!

[mjlbach]

Fixes the problem for me!

[clason]

Yes, works really smoothly now. Thank you!

[charlie39]

What about Linux and macos?