True/False being reported as undefined

[brettcannon]

microsoft/vscode-python#3274

This has been reported by at least 3 separate people (see the duplicate issue to the one above).

[ccaspers]

Probably related: the same happens for me with None.

from typing import Generator


def fun() -> Generator[int, None, None]:
    yield from range(10)

[KevOrr]

Fwiw I get the same with None being used as an argument to a type annotation

MWE: (see edit)

'None' (the string) produces the same effect:

True and False being substituted there don't produce the same effect:

EDIT: Even more minimal MWE:

[jakebailey]

I have a fix for the None issue (see the followup comments in #429), but that's unrelated to the True/False issue. I've opened another issue (#442) to track the None bug separately from the True/False so we aren't mixing two different problems in this thread.

The True/False issue is likely some racy behavior where we do analysis before the builtins are finished (something I've been unable to reproduce).

[jarshwah]

I reported the original issue on the vscode-python tracker. I can reliably reproduce this issue every time I open my (fairly large) python 2.7 project. The True/False identifiers are only marked use before ref once the project has finished being analyzed. That said, this particular project is also prone to maxing out CPU and constantly growing RAM (with the language server) so perhaps those two things are related.

[jakebailey]

Is your project open source where we could test it out? I personally have only been able to make it happen once, and never again. I've tested on a number of large repos to no avail.

My current thinking is that there's some race or bug that's allowing analysis to happen before we analyze the builtins themselves. For a large project which takes a long time to work, it may exasperate the issue (and allow us to find it).

Any more info you could give about your platform would be useful (OS, are you using a virtualenv, where is the virtualenv stored if so, etc).

[jarshwah]

Sorry, it's not open source, but I can try to provide as many details as you need.

OS: OSX 10.13.6
Python Version: built with pyenv

Python 2.7.14 (default, Mar 11 2018, 00:52:26)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin

Virtualenv: pyenv-virtualenv and using .python-version in local directory to point to it.

Relevant vscode settings:

"python.pythonPath": "/Users/josh/.pyenv/versions/k3env/bin/python",
"python.autoComplete.extraPaths": [
        "${workspaceRoot}/kogan3/apps/",
        "${workspaceRoot}/kogan3/",
    ],
"python.venvPath": "/Users/josh/.virtualenvs/",

An important detail may be our directory structure which is non-standard. This is a django project. We contain our apps within an internal apps subdirectory which we add to our PYTHONPATH in manage.py. See the structure below (heavily pruned):

.
├── kogan3  <-- our project path
│   ├── apps  <-- added to PYTHONPATH
│   │   ├── address
│   │   ├── auditing
│   │   ├── banners
│   │   ├── core
│   │   ├── k3
│   │   │   ├── cart
│   │   │   ├── collections
│   │   │   ├── customers
│   │   ├── knowledge
│   ├── js-src
│   │   ├── reactcomponents
│   ├── kogan3
│   │   ├── settings
│   │   ├── swagger
│   │   └── tests
│   ├── node_modules
│   ├── site_media
│   │   ├── admin
│   │   ├── grappelli
│   ├── staticfiles
│   │   ├── admin
│   │   ├── email_templates
│   │   ├── fonts
│   │   ├── js-build
│   ├── templates
│   └── webpack
├── requirements

In this way, we refer to our apps like:

from kogan3.settings import base
from k3.cart import models
from address import models

Some files use from __future__ import absolute_import while others do not.

Please let me know if there's any further information that would be beneficial.

[jarshwah]

Could this possibly be caused by the python-future library, which defines a module called builtins? I'm using this package in my project, and as I was checking the AnalysisLog I noticed a lot of entries regarding builtins.

I can provide the AnalysisLog if you think that could be useful.

[jakebailey]

I was able to get it (exactly once) without importing anything, so I don't think it's necessarily one library or another. Installing the future package and importing a few different ways doesn't trigger anything either.

Right now, I think I'm just going to do something like in #429 and hardcode a few known-defined names until we can figure out the actual cause. The code that produces these messages is going to get completely remove as we simplify the codebase, so the bug may also disappear as well.

[jakebailey]

@jarshwah If you've been able to reliably get True/False to show as use-before-def, would you mind trying out our daily build to see if the issue goes away? I added a temporary fix in #464 (not yet in stable). To move to the daily build, you can add this to your VS Code settings:

"python.analysis.downloadChannel": "daily"

If you want to go back to stable, you can remove that line and delete the folders with names starting with languageServer inside of the Python VS Code extension's directory.

[jarshwah]

Unfortunately the language server is crashing for me now, as of version: languageServer.0.1.69

Logs for version languageServer.0.1.71 below:

[Error - 5:37:55 PM] System.InvalidCastException: Unable to cast object of type 'Microsoft.PythonTools.Analysis.Values.BuiltinFunctionInfo' to type 'Microsoft.PythonTools.Analysis.Values.BuiltinClassInfo'.
   at Microsoft.PythonTools.Analysis.PythonAnalyzer.GetBuiltinType(IPythonType type)
   at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()
   at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
   at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
   at Microsoft.PythonTools.Analysis.Values.Mro..ctor(IEnumerable`1 values)
   at Microsoft.PythonTools.Analysis.Values.BuiltinClassInfo.get_Mro()
   at Microsoft.PythonTools.Analysis.Values.Mro.Recompute()
   at Microsoft.PythonTools.Analysis.Values.ClassInfo.SetBases(IEnumerable`1 bases)
   at Microsoft.PythonTools.Analysis.ClassAnalysisUnit.AnalyzeWorker(DDG ddg, CancellationToken cancel)
   at Microsoft.PythonTools.Analysis.Analyzer.DDG.Analyze(Deque`1 queue, CancellationToken cancel, Action`1 reportQueueSize, Int32 reportQueueInterval)
   at Microsoft.PythonTools.Analysis.PythonAnalyzer.AnalyzeQueuedEntries(CancellationToken cancel)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.GroupAnalysis.Analyze(CancellationToken cancel)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.HandleAnalyzable(IAnalyzable item, AnalysisPriority priority, CancellationToken cancellationToken)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.ConsumerLoop()
Hover in file:///Users/josh/dev/K3/kogan3/apps/some_gateway/models.py at (522, 28)
Waiting for parsing to complete.
Parsing complete. Waiting for analysis entries to enqueue.
Enqueue complete. Waiting for analysis to complete.
[Error - 5:42:11 PM] Request textDocument/definition failed.
  Message: PriorityProducerConsumer`1 instance is disposed
Object name: 'PriorityProducerConsumer`1'.
  Code: -32000 
   at Microsoft.PythonTools.Analysis.Infrastructure.DisposeToken.ThrowIfDisposed()
   at Microsoft.PythonTools.Analysis.Infrastructure.PriorityProducerConsumer`1.Produce(T value, Int32 priority)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.ExecuteInQueueAsync(Func`2 function, AnalysisPriority priority)
   at Microsoft.Python.LanguageServer.Implementation.Server.WaitForCompleteAnalysisWorker(CancellationToken cancellationToken)
   at Microsoft.Python.LanguageServer.Implementation.Server.FindReferences(ReferencesParams params, CancellationToken cancellationToken)
   at Microsoft.Python.LanguageServer.Implementation.Server.GotoDefinition(TextDocumentPositionParams params, CancellationToken cancellationToken)
   at Microsoft.Python.LanguageServer.Implementation.LanguageServer.GotoDefinition(JToken token, CancellationToken cancellationToken)
Getting hover for Order
Waiting for parsing to complete.
Parsing complete. Waiting for analysis entries to enqueue.
Enqueue complete. Waiting for analysis to complete.
[Error - 5:42:13 PM] Request textDocument/definition failed.
  Message: PriorityProducerConsumer`1 instance is disposed
Object name: 'PriorityProducerConsumer`1'.
  Code: -32000 
   at Microsoft.PythonTools.Analysis.Infrastructure.DisposeToken.ThrowIfDisposed()
   at Microsoft.PythonTools.Analysis.Infrastructure.PriorityProducerConsumer`1.Produce(T value, Int32 priority)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.ExecuteInQueueAsync(Func`2 function, AnalysisPriority priority)
   at Microsoft.Python.LanguageServer.Implementation.Server.WaitForCompleteAnalysisWorker(CancellationToken cancellationToken)
   at Microsoft.Python.LanguageServer.Implementation.Server.FindReferences(ReferencesParams params, CancellationToken cancellationToken)
   at Microsoft.Python.LanguageServer.Implementation.Server.GotoDefinition(TextDocumentPositionParams params, CancellationToken cancellationToken)
   at Microsoft.Python.LanguageServer.Implementation.LanguageServer.GotoDefinition(JToken token, CancellationToken cancellationToken)

[jarshwah]

FWIW - I've done some c# in my time. So if there are any decent "getting started with contributing" guides and some tests you'd like me to run, I'd be happy to do so.

[crucialfelix]

I just tested with "daily" and I got the same Error as @jarshwah posted.

I then removed that config, but the error still happens.

If you want to go back to stable, you can remove that line and delete the folders with names starting with languageServer inside of the Python VS Code extension's directory.

There are no such folders in extensions:

ls ~/.vscode/extensions | grep language
(nothing)

I see various items in CachedData.

Code ❯ find "$PWD" -type f | grep languageServer
/Users/crucialfelix/Library/Application Support/Code/CachedData/5f24c93878bd4bc645a4a17c620e2487b11005f9/languageServer-bcc19ccf182adfdbe4f4b9f5b50f3905.code
/Users/crucialfelix/Library/Application Support/Code/CachedData/5f24c93878bd4bc645a4a17c620e2487b11005f9/languageServerFolderService-f0562a6a373bc9dca87a783076d5b348.code
/Users/crucialfelix/Library/Application Support/Code/CachedData/5f24c93878bd4bc645a4a17c620e2487b11005f9/languageServerSurveyBanner-57fcf0ee6d5f224e3f5142f34fb7ab4a.code
/Users/crucialfelix/Library/Application Support/Code/CachedData/5f24c93878bd4bc645a4a17c620e2487b11005f9/languageServerHashes-1f4106225ddbbe1987b48e5e1468a337.code
/Users/crucialfelix/Library/Application Support/Code/CachedData/5f24c93878bd4bc645a4a17c620e2487b11005f9/languageServerPackageRepository-5df63595ebc0344cba8e97a6ad4d9972.code
/Users/crucialfelix/Library/Application Support/Code/CachedData/5f24c93878bd4bc645a4a17c620e2487b11005f9/languageServerPackageService-e7d883d1659623cea2d6939c4393b442.code
/Users/crucialfelix/Library/Application Support/Code/CachedData/bc24f98b5f70467bc689abf41cc5550ca637088e/languageServer-bcc19ccf182adfdbe4f4b9f5b50f3905.code
/Users/crucialfelix/Library/Application Support/Code/CachedData/bc24f98b5f70467bc689abf41cc5550ca637088e/languageServerFolderService-f0562a6a373bc9dca87a783076d5b348.code
/Users/crucialfelix/Library/Application Support/Code/CachedData/bc24f98b5f70467bc689abf41cc5550ca637088e/languageServerSurveyBanner-57fcf0ee6d5f224e3f5142f34fb7ab4a.code
/Users/crucialfelix/Library/Application Support/Code/CachedData/bc24f98b5f70467bc689abf41cc5550ca637088e/languageServerHashes-1f4106225ddbbe1987b48e5e1468a337.code
/Users/crucialfelix/Library/Application Support/Code/CachedData/bc24f98b5f70467bc689abf41cc5550ca637088e/languageServerPackageRepository-5df63595ebc0344cba8e97a6ad4d9972.code
/Users/crucialfelix/Library/Application Support/Code/CachedData/bc24f98b5f70467bc689abf41cc5550ca637088e/languageServerPackageService-e7d883d1659623cea2d6939c4393b442.code

[jarshwah]

@crucialfelix you need to delete the language server folder from the python extension folder:

~/.vscode/extensions/ms-python.python-2018.11.0/languageServer.0.1.71

[jakebailey]

If you'd like to look yourself, we have instructions in CONTRIBUTING.md which should work on any platform.

Since this is unrelated to the True/False issue, I've created #470 for that discussion.

[jakebailey]

@jarshwah @crucialfelix The daily build now includes a fix for #470 (0.1.72.0), if you'd like to retest my temporary fix.

[jarshwah]

Looks like that has done it, thanks!

[jakebailey]

Great!

I'm going to close this issue for now as the code that produces these warnings is slated to be removed/redone during our ongoing rewrite. We'll revisit this later just to make sure this sort of thing doesn't happen again with the new code.