Better progress reporting #96

MikhailArkhipov · 2018-09-14T18:24:41Z

Fixes #94

Make separate class for progress reporting
Do not use 'estimate work' and rather track specific events
Make reporting thread safe so we don't end up with multiple competing reporting tasks

AlexanderSher · 2018-09-14T19:39:40Z

src/LanguageServer/Impl/Definitions/ILogger.cs

@@ -19,5 +19,6 @@
 namespace Microsoft.Python.LanguageServer {
    public interface ILogger {
        void TraceMessage(IFormattable message);
+        void TraceMessage(string message);


This has an issue of creating strings (lots of strings) even when logging is switched off or doesn't meet the required severity. It creates noticeable performance impact. If we need to get formatted strings from resources and can't use IFormattable, we can add TraceMessageFormat(string format, params object[] parameters) and do formatting only when logging is enabled.

AlexanderSher · 2018-09-14T19:41:21Z

src/LanguageServer/Impl/Resources.resx

@@ -117,6 +117,12 @@
  <resheader name="writer">
    <value>System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
  </resheader>
+  <data name="AnalysisProgress_MultipleItemsRemaining" xml:space="preserve">


If those strings go directly to the client, how do we choose the culture?

It is to be determined.... Unfortunately, LSP does not specify locale, so we'll have to add some custom notification. Which reminds me to file #98

Can't we have those strings on VSC side and provide only numbers?

VSC is not the only client. LSP is public and supported by Eclipse, Vim, Emacs and more. So there is actually a plan to try LS with one of those.

Ok, but isn't this part of the LS protocol is python specific and we will have to add something to the client-side anyway?

Something like python/setLocale. If client doesn't send it, we'll just use English. Also, clients typically expect string in exceptions which they display in UI. We throw with descriptive message when we cannot rename, for example.

AlexanderSher · 2018-09-14T21:56:41Z

src/LanguageServer/Impl/Implementation/AnalysisProgressReporter.cs

+        private void UpdateProgressMessage() {
+            if(_activeAnalysis.Count > 0) {
+                _progress = _progress ?? _progressService.BeginProgress();
+                _progress.Report(_activeAnalysis.Count == 1


So this is more like a status message rather than a progress, cause _activeAnalysis.Count may actually go up, correct?

Yeah. This is how progresss is reported in VSC - via spinner and message in the status bar. I am not 100% about count but since analysis is recursive and event may fire from threads, I better be safe

AlexanderSher · 2018-09-14T22:01:02Z

src/LanguageServer/Impl/Implementation/AnalysisProgressReporter.cs

+            lock (_lock) {
+                if (_activeAnalysis.TryGetValue(e.uri, out var count)) {
+                    if (count > 1) {
+                        _activeAnalysis[e.uri]--;


PythonAnalyzer.AnalyzeQueuedEntries may exit before raising OnAnalysisComplete. While this one may happen only when server is disposed, I'm not sure these events (OnAnalysisQueued and OnAnalysisComplete) are actually guaranteed to be raised in pairs.

Interesting. This may explain stale messages in the status bar. I will check it out.

OK then, back to monitoring queue. The important part is to make sure we do NOT display progress when nothing is being analyzed.

sneridagh · 2018-09-18T07:36:57Z

Can we have a release? The problem that this fix is very annoying...

MikhailArkhipov · 2018-09-18T17:32:48Z

@sneridagh - try insider builds starting with 9/19 build, see https://github.com/Microsoft/vscode-python/blob/master/CONTRIBUTING.md#development-build

fgrcon · 2018-09-24T06:38:43Z

@MikhailArkhipov - i installed the insider beta right now, but it does not help - still get stuck with "Analyzing workspace, 3467 items left" number increasing .....

sneridagh · 2018-09-24T11:51:39Z

using 20/9 build:
Same behavior, the issue persists.
One CPU locked, fans spinning, Memory usage 2GB.

sneridagh · 2018-09-24T15:03:52Z

@MikhailArkhipov any insight?

MikhailArkhipov · 2018-09-24T19:11:49Z

@sneridagh - Unfortunately, new builds are not available yet, we are changing build process in order to better support dynamic updates of the LS in the VS Code extension. You can try and build it yourself or wait couple of days until we settle on URLs and versioning.

fgrcon · 2018-09-25T07:04:49Z

@MikhailArkhipov where can I dowload an older (last stable) extension?

yzhang-gh · 2018-09-25T11:17:28Z

Hope it can be solved soon 🥺

@fgrcon Maybe try uninstalling the current one, restarting vscode, and then installing from the marketplace.

fgrcon · 2018-09-25T15:31:39Z

Hope it can be solved soon 🥺

@fgrcon Maybe try uninstalling the current one, restarting vscode, and then installing from the marketplace.

@neilsustc : that'is what I tried, but i have no clue where to find previous releases ...

sneridagh · 2018-09-26T07:58:54Z

@MikhailArkhipov Thanks for your patience and support... but somehow I'm unable to build it on MACOS. I even tried to follow the steps by hand but multiple points of failure... The build process just doesn't build on it. I cannot spend more time on this, I will wait until October... it's a pity that the release process is that tight and cannot be bent when it's required. Something really uncommon in OSS.

What it really concerns me is that we haven't been able to test the patch properly... what if it doesn't work? We will have to wait until November and hope that all is fine?

Also, I don't understand yet the source of the problem. There's a setting that states clearly:

  // Only show errors and warnings for open files rather than for the entire workspace.
  "python.analysis.openFilesOnly": true,

I don't want my entire workspace to be analyzed, and it seems that it's what is happening. Is this setting working as expected?

By the way, I workaround it by (literally)

$ rm -rf ~/.vscode/extensions/ms-python.python-2018.8.0/languageServer

and I have at the end the same outcome but without my CPU spinning like crazy during all day.

DonJayamanne · 2018-09-27T00:41:24Z

@sneridagh
If you would like to test the Python extension with the latest version of the language server then please:

Install the development version of the Python Extension from here https://pvsc.blob.core.windows.net/extension-builds/ms-python-insiders.vsix
@MikhailArkhipov I don't know what version of the language server they need.
The version of Language Server that will get downloaded by the dev version of the extension is 0.1.24

sneridagh · 2018-09-27T07:14:25Z

@DonJayamanne I already tried this build (2018.8.1-beta (20 September 2018) and as confirmed by @MikhailArkhipov, it doesn't contain the fix or either since it doesn't have changelog entry, it doesn't work.

yzhang-gh · 2018-09-27T09:33:58Z

@DonJayamanne Sorry for digression. Could you please also publish (or package) a new version of your vscodeJupyter extension? Some PRs were merge many monthes ago but we still cannot benefit from them now 😥.
~~(I tried to package it by myself but failed for knowing little about webpack and some package version conflicts(?))~~
I finally figured it out

MikhailArkhipov · 2018-09-27T15:20:02Z

20th was still an old build. We published update on 28th.

fgrcon · 2018-09-28T08:59:27Z

@MikhailArkhipov interesting: you are able to publish in the future ? ;-)
but https://pvsc.blob.core.windows.net/extension-builds/ms-python-insiders.vsix is still the same from 9/20!

MikhailArkhipov · 2018-09-28T16:12:28Z

@fgrcon - Right but it downloads proper LS. VSIX itself was not an issue, content of the storage blob was. I guess it was 27th. :-)

sneridagh · 2018-09-28T17:26:27Z

@MikhailArkhipov The I would say that problem is still there. Probably you can here my fans blowing from where you are.

Ok, my workspace is huge, but it's not what the vast majority of people will have? In large projects where hundreds of eggs are used, I mean...

It should be something that has changed recently, even Jedi gets stuck and make my CPUs spin going loco. Before summer this didn't happen. Please, I'm considering exiting VSCode in favor of other IDE for Python because of it.

I would really want to help pinpoint what the cause is, if I'm given directions. I cannot see anymore the enable traceback log setting. Did you remove it?

sneridagh · 2018-09-28T17:46:48Z

@MikhailArkhipov FYI I was giving it another try, then the process blew and rebooted my mac in my face.

AlexanderSher · 2018-09-28T17:57:09Z

Is it a publicly available source code?

MikhailArkhipov · 2018-09-28T19:12:20Z

@sneridagh - With huge workspace it is a different problem.

The bug was that analysis reporting was stuck, i.e. reporting items outstanding while actually there were none. With huge projects analysis may take a long time. That does not mean though that product is not usable. You don't have to wait for everything to complete.

If Jedi is stuck it means even Python engine takes a long time to build and walk ASTs. While C# is more performant in general, it is not a panacea. It won't be orders of magnitude faster.

sneridagh · 2018-09-28T21:03:14Z

@MikhailArkhipov it gets stuck as well, in fact it's a combination of both factors. The analysis, being huge, never ends, the process eats all the memory available. The message in the status bar "X items remaining" doesn't go away.

TomFaulkner · 2018-10-01T17:10:44Z

While it isn't unusable, the whole computer slows to a crawl while the Python language server is analyzing. A 2016 MacBook Pro has the Microsoft Python Language Server taking nearly two of the four cores to itself, and half of the 16gb of RAM the system has.

Installing this https://pvsc.blob.core.windows.net/extension-builds/ms-python-insiders.vsix seems to have resolved the issue.

sneridagh · 2018-10-02T11:28:45Z

@MikhailArkhipov @DonJayamanne Please provide a setting for disable completely both Jedi and languageServer analyzers. I prefer not to have Intellisense at all that having my CPU fans blowing continuously.

@TomFaulkner it works for a while... until then it starts again (even after some hours).

@MikhailArkhipov now it does it even partitioning my workspace into smaller parts (not loading the big ones, eggs sources). It stays calmed, until it begins again. Just killed it manually (the languageServer process) and it stays calm again for a while. Don't know why it causes to start to get crazy again.

TomFaulkner · 2018-10-02T21:18:58Z

@sneridagh, you can remove the Python extension, or disable it. That is what I did prior to searching out the issue and using the insider build above. (Which is so far working.)

sneridagh · 2018-10-03T07:23:49Z

@TomFaulkner then I'll loose linting, etc... I only want to get rid of intellisense.

TomFaulkner · 2018-10-03T16:46:31Z

Sadly, yes, that is the impact.

fgrcon · 2018-11-05T14:44:09Z

sadly giving up!
the latest release still eats up the whole memory within minutes and everything gets stuck.

how to get back to working release (i think early summer this year ) - I need debugging, linting & outline but can live without greedy intellisense, if there is no other way.

TomFaulkner · 2018-11-05T16:11:35Z

sadly giving up!
the latest release still eats up the whole memory within minutes and everything gets stuck.

how to get back to working release (i think early summer this year ) - I need debugging, linting & outline but can live without greedy intellisense, if there is no other way.

@fgrcon, I was using the beta release to get around this issue. Then after an update it got replaced and I was back to this problem. However, I noticed there was an update available for the extension, and that update resolved things. I hope this helps you, I was ready to switch as well.

sneridagh · 2018-11-15T08:51:51Z

@TomFaulkner @fgrcon About to give up as well... Returning either to Atom or Sublime. @fgrcon where did you get the update? I'm on 2018.10.1 (09 Nov 2018) (it seems).

/cc @MikhailArkhipov Again, please provide a way to disable intellisense (either via Jedi or PLS) in the plugin, for those that we don't care about it, but we do about the other things the plugin provides. Thanks.

Better progress reporting

MikhailArkhipov added 2 commits September 14, 2018 11:21

Better progress reporting

e5fcc19

Remove unused field

5156180

AlexanderSher reviewed Sep 14, 2018

View reviewed changes

MikhailArkhipov added 4 commits September 14, 2018 12:44

PR feedback

a2b7b93

Enable tests

35c494e

Enable test

09bddb5

Make interface public

5dc3c80

AlexanderSher reviewed Sep 14, 2018

View reviewed changes

Back to queue tracking

7380582

AlexanderSher approved these changes Sep 17, 2018

View reviewed changes

MikhailArkhipov merged commit 2fbe616 into microsoft:master Sep 17, 2018

jakebailey pushed a commit to jakebailey/python-language-server that referenced this pull request Nov 1, 2019

Merge pull request microsoft#96 from MikhailArkhipov/master

bb0954d

Better progress reporting

Better progress reporting #96

Better progress reporting #96

Conversation

MikhailArkhipov commented Sep 14, 2018 • edited Loading

AlexanderSher Sep 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexanderSher Sep 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexanderSher Sep 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sneridagh commented Sep 18, 2018

MikhailArkhipov commented Sep 18, 2018 • edited Loading

fgrcon commented Sep 24, 2018

sneridagh commented Sep 24, 2018

sneridagh commented Sep 24, 2018

MikhailArkhipov commented Sep 24, 2018

fgrcon commented Sep 25, 2018

yzhang-gh commented Sep 25, 2018

fgrcon commented Sep 25, 2018

sneridagh commented Sep 26, 2018 • edited Loading

DonJayamanne commented Sep 27, 2018

sneridagh commented Sep 27, 2018 • edited Loading

yzhang-gh commented Sep 27, 2018 • edited Loading

MikhailArkhipov commented Sep 27, 2018

fgrcon commented Sep 28, 2018 • edited Loading

MikhailArkhipov commented Sep 28, 2018

sneridagh commented Sep 28, 2018

sneridagh commented Sep 28, 2018

AlexanderSher commented Sep 28, 2018

MikhailArkhipov commented Sep 28, 2018 • edited Loading

sneridagh commented Sep 28, 2018

TomFaulkner commented Oct 1, 2018 • edited Loading

sneridagh commented Oct 2, 2018

TomFaulkner commented Oct 2, 2018

sneridagh commented Oct 3, 2018

TomFaulkner commented Oct 3, 2018

fgrcon commented Nov 5, 2018

TomFaulkner commented Nov 5, 2018

sneridagh commented Nov 15, 2018

MikhailArkhipov commented Sep 14, 2018 •

edited

Loading

AlexanderSher Sep 14, 2018 •

edited

Loading

AlexanderSher Sep 14, 2018 •

edited

Loading

AlexanderSher Sep 14, 2018 •

edited

Loading

MikhailArkhipov commented Sep 18, 2018 •

edited

Loading

sneridagh commented Sep 26, 2018 •

edited

Loading

sneridagh commented Sep 27, 2018 •

edited

Loading

yzhang-gh commented Sep 27, 2018 •

edited

Loading

fgrcon commented Sep 28, 2018 •

edited

Loading

MikhailArkhipov commented Sep 28, 2018 •

edited

Loading

TomFaulkner commented Oct 1, 2018 •

edited

Loading