Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge persistence work into master #1482

Merged
merged 79 commits into from Sep 16, 2019

Conversation

@MikhailArkhipov
Copy link
Member

commented Aug 26, 2019

We accumulated many bug fixes in db branch so it will be useful to merge it to master. Will turn off persistence service before merging.

Fixes #1531.
Fixes #1534
Fixes #1550.

MikhailArkhipov and others added 30 commits Jun 20, 2019
Persistent analysis, part I (#1224)
* Remove old qualified name

* Node storage

* Class and scope to use AST map

* Library analysis

* Fix SO

* Keep small AST with imports

* AST reduction

* Final field

* Initial

* Reload

* Ignore post-final requests

* Drop AST

* Remove local variables

* Test fixes

* Fix overload match

* Tests

* Add locks

* Remove local variables

* Drop file content to save memory

* Cache PEP hints

* Recreate AST

* Fix specialization

* Fix locations

* usings

* Test fixes

* Add options to keep data in memory

* Fix test

* Fix lambda parameters

* Fix argument set
Fix global scope node

* Fix overload doc

* Fix stub merge errors

* Fix async issues

* Undo some changes

* Fix test

* Fix race condition

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback
Merge master into database branch (#1312)
* Remove old qualified name

* Node storage

* Class and scope to use AST map

* Library analysis

* Fix SO

* Keep small AST with imports

* AST reduction

* Final field

* Initial

* Reload

* Ignore post-final requests

* Drop AST

* Remove local variables

* Test fixes

* Fix overload match

* Tests

* Add locks

* Remove local variables

* Drop file content to save memory

* Cache PEP hints

* Recreate AST

* Fix specialization

* Fix locations

* usings

* Test fixes

* Add options to keep data in memory

* Fix test

* Fix lambda parameters

* Fix argument set
Fix global scope node

* Fix overload doc

* Fix stub merge errors

* Fix async issues

* Undo some changes

* Fix test

* Fix race condition

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback

* Avoid empty node in empty global scope (#1226)

* Add null checks

* Avoid null node in empty global scope

* Remove unrelated change

* Lock content since it is locked in other cases (#1227)

* Fix addBrackets setting (#1241)

* Fix addBrackets

* Remove unrelated change

* Usings

* Handle library analysis better (#1246)

* Trying to augment argument set with Evaluator and Expression context (#1259)

* Moving anyStr to its own method to prevent FromTypeVar from being called with an ArgumentSet that has no context

* Trying to augment argument set with context wherever possible

* Setting expression in argument set, removing null

* Renaming to WithoutContext and updating comment

* Check for null args in PythonFunctionOverload

* When returning in init method, report a diagnostic error (#1261)

* When returning in init method, report a diagnostic error

* Adding check and tests for returning None before adding error for a return statement in the init function

* Cleaning up tests

* Give Diagnostic messages on improper usage of Generic (#1248)

* Adding diagnostic message for improper use Generic.

* Adding tests to make sure no diagnostics on valid uses of Generic

* Adding diagnostic error on binary operations with incompatible types (#1254)

* Adding diagnostic error for binary operations with incompatible types - e.g 5 + 'str'

* When NewType is called and the first argument is not a string, make a diagnostic message (#1260)

* Adding diagnostic error if the user calls NewType with the first arg not of string type

* Fix typo in TROUBLESHOOTING.md (#1285)

* Multiple analysis fixes (#1297)

Multiple analysis fixes.
- Fixes #1151: Reliable way to determine final analysis of a module
- Fixes #1201: NRE in SymbolCollector.AddProperty
- Fixes #1228: NRE in TryFindMissingDependencies
- Fixes #1294: Handle valid case for reloading AST
- Fixes #1295: FindReferences doesn't work on package init members
- Fixes #1296: DependencyResolver graph misses intermediate imports
- Part of the fix for #1174: All references are not listed when using ms language server (fixes in analysis itself are also required for this one)

* Rework search path resolution (#1289)

* first working path classfier, breaks existing path code for now

* inherit Comparer instead of implementing both

* don't do path depth comparison, preserve user ordering as before

* switch to using new path classification code in main module resolution

* fix typo

* clean up comment about paths in Server.cs

* use String.Split instead of regex when parsing script output

* test ordering of user provided paths

* add new normalize and trim helper, check preconditions as debug asserts rather than commenting on them

* use Split extension, don't MaybeEnumerate paths

* No diagnostic message when Generic is called with no args (#1305)

* No diagnostic message when Generic is called with no args

* When a class inherits from something that is not a class, give a diagnostic error (#1277)

* When analyzing class definitions, give diagnostic if bases are not all class types

* PR feedback
Treat persistent module as regular specialized (#1334)
* Remove old qualified name

* Node storage

* Class and scope to use AST map

* Library analysis

* Fix SO

* Keep small AST with imports

* AST reduction

* Final field

* Initial

* Reload

* Ignore post-final requests

* Drop AST

* Remove local variables

* Test fixes

* Fix overload match

* Tests

* Add locks

* Remove local variables

* Drop file content to save memory

* Cache PEP hints

* Recreate AST

* Fix specialization

* Fix locations

* usings

* Test fixes

* Add options to keep data in memory

* Fix test

* Fix lambda parameters

* Fix argument set
Fix global scope node

* Fix overload doc

* Fix stub merge errors

* Fix async issues

* Undo some changes

* Fix test

* Fix race condition

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback

* PR feedback

* Restore persistence + update test

* Better handle persistent module in dependencies

* Undo

* Fix merge issue

* Remove debug code
Store location of module members in the database (#1339)
* Remove old qualified name

* Node storage

* Class and scope to use AST map

* Library analysis

* Fix SO

* Keep small AST with imports

* AST reduction

* Final field

* Initial

* Reload

* Ignore post-final requests

* Drop AST

* Remove local variables

* Test fixes

* Fix overload match

* Tests

* Add locks

* Remove local variables

* Drop file content to save memory

* Cache PEP hints

* Recreate AST

* Fix specialization

* Fix locations

* usings

* Test fixes

* Add options to keep data in memory

* Fix test

* Fix lambda parameters

* Fix argument set
Fix global scope node

* Fix overload doc

* Fix stub merge errors

* Fix async issues

* Undo some changes

* Fix test

* Fix race condition

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback

* PR feedback

* Restore persistence + update test

* Better handle persistent module in dependencies

* Undo

* Add location converter abstraction

* Store member location

* Basic locations test

* Navigation

* Add test

* Update baselines

* Port changes from dbtype
Multiple fixes and additional tests for persistence (#1372)
* Remove old qualified name

* Node storage

* Class and scope to use AST map

* Library analysis

* Fix SO

* Keep small AST with imports

* AST reduction

* Final field

* Initial

* Reload

* Ignore post-final requests

* Drop AST

* Remove local variables

* Test fixes

* Fix overload match

* Tests

* Add locks

* Remove local variables

* Drop file content to save memory

* Cache PEP hints

* Recreate AST

* Fix specialization

* Fix locations

* usings

* Test fixes

* Add options to keep data in memory

* Fix test

* Fix lambda parameters

* Fix argument set
Fix global scope node

* Fix overload doc

* Fix stub merge errors

* Fix async issues

* Undo some changes

* Fix test

* Fix race condition

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback

* PR feedback

* Restore persistence + update test

* Better handle persistent module in dependencies

* Undo

* Add location converter abstraction

* Store member location

* Fix merge issue

* Basic locations test

* Navigation

* Add test

* Update baselines

* Type restore - initial

* Remove debug code

* Partial

* Fix stub merge

* Various model fixes

* Improve module handling

* Qualified name improvements

* Fix unbound case

* Improve stub merge

* Fix qualified names of typing

* Handle stub-only modules

* Add tests for io, re and sys

* Better handle named tuple

* Handle module circular references

* Handle else in platform and version clauses + handle it in symbol collector

* Formatting

* Fix walk of multi-level if statement

* Unify package search in imports

* Fix tests

* Undo change

* Port changes from dbtype

* Partial

* Named tuple support

* Baseline updates

* Debug code

* Support types

* Properly compare class member declaring type

* Nested classes and functions persistence

* Undo debug

* Fix numpy restore

* Baselines

* Fix tests

* Update AnyStr test reflecting changes to AnyStr behavior

* Exclude baselines from git diff

* Fix gitattr

* Move git setting to root

* Try no path

* Add RDT count to the analysis_complete event (#1396)

* Add RDT count to the analysis_complete event

* Use property instead of method

* Test fixes

* Undo change

* Additional stub merge fixes

* Baseline updates

* Fix goto def

* Protect specific type creation

* Track documentaton source

* More reliable tests + simplification

* Prevent crashes when the AST happens to be null (#1405)

* Remove all references to dropped modules and force GC on reload (#1402)

* First attempt at ensuring moduled dropped during reload are GC'd

* Make sure AST is set after content is cleared

* More tests passing

* ModuleWalker updates

* Also don't merge against specialized modules, do builtin _astMap reset before reanalyzing instead of after any analysis

* Don't call dispose directly, it's done elsewhere and has no effect

* Force GC via normal session instead of waiting for a hardcoded time

* Un-refactor function since it's not being called externally anymore

* Formatting/usings

* PR feedback

* Undo IsBuiltin, some things continued to be referenced with that changed

* Move reload logic down into PythonAnalyzer, remove added public inferface items

* Mode AddReference to PythonType as override

* Remove dead code, make all ResetAnalyzer calls full

* Typo

* Cleanup

* Basic classification

* Fix merge error

* Module unique id fixes

* Stricted check to save analysis

* Revert "Fix tests"

This reverts commit 247a8c3.

* Revert "Unify package search in imports"

This reverts commit 67fed10.

* Don't clear scope variables with inner classes

* Fix typo
Add persistence tests for 60+ system modules (#1429)
* Remove old qualified name

* Node storage

* Class and scope to use AST map

* Library analysis

* Fix SO

* Keep small AST with imports

* AST reduction

* Final field

* Initial

* Reload

* Ignore post-final requests

* Drop AST

* Remove local variables

* Test fixes

* Fix overload match

* Tests

* Add locks

* Remove local variables

* Drop file content to save memory

* Cache PEP hints

* Recreate AST

* Fix specialization

* Fix locations

* usings

* Test fixes

* Add options to keep data in memory

* Fix test

* Fix lambda parameters

* Fix argument set
Fix global scope node

* Fix overload doc

* Fix stub merge errors

* Fix async issues

* Undo some changes

* Fix test

* Fix race condition

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback

* PR feedback

* Restore persistence + update test

* Better handle persistent module in dependencies

* Undo

* Add location converter abstraction

* Store member location

* Fix merge issue

* Basic locations test

* Navigation

* Add test

* Update baselines

* Type restore - initial

* Remove debug code

* Partial

* Fix stub merge

* Various model fixes

* Improve module handling

* Qualified name improvements

* Fix unbound case

* Improve stub merge

* Fix qualified names of typing

* Handle stub-only modules

* Add tests for io, re and sys

* Better handle named tuple

* Handle module circular references

* Handle else in platform and version clauses + handle it in symbol collector

* Formatting

* Fix walk of multi-level if statement

* Unify package search in imports

* Fix tests

* Undo change

* Port changes from dbtype

* Partial

* Named tuple support

* Baseline updates

* Debug code

* Support types

* Properly compare class member declaring type

* Nested classes and functions persistence

* Undo debug

* Fix numpy restore

* Baselines

* Fix tests

* Update AnyStr test reflecting changes to AnyStr behavior

* Exclude baselines from git diff

* Fix gitattr

* Move git setting to root

* Try no path

* Test fixes

* Undo change

* Additional stub merge fixes

* Baseline updates

* Fix goto def

* Protect specific type creation

* Track documentaton source

* More reliable tests + simplification

* Typo

* Cleanup

* Basic classification

* Fix merge error

* Module unique id fixes

* Stricted check to save analysis

* Revert "Fix tests"

This reverts commit 247a8c3.

* Revert "Unify package search in imports"

This reverts commit 67fed10.

* Don't clear scope variables with inner classes

* Fix typo

* Many new tests

* Fix collections test

* Fix CTypes

* Initial

* Update test

* Merge issues

* Fix CTypes again

* Fix null bases

* Tell between class members with/without self better

* TypeVar support

* Add bound/covariant

* Fix Enum reassignments

* Fix Random

* Fix import * over local declarations (Socket)

* Move interface

* Fix reference search

* Enable asyncio test

* More tests

* Enable dataclasses

* Add inspect and gzip

* More tests

* Add setting

* Add handling of import position relative to the variable

* Caching level changes

* Update baselines
Persistence of generics and named tuples (#1459)
* Remove old qualified name

* Node storage

* Class and scope to use AST map

* Library analysis

* Fix SO

* Keep small AST with imports

* AST reduction

* Final field

* Initial

* Reload

* Ignore post-final requests

* Drop AST

* Remove local variables

* Test fixes

* Fix overload match

* Tests

* Add locks

* Remove local variables

* Drop file content to save memory

* Cache PEP hints

* Recreate AST

* Fix specialization

* Fix locations

* usings

* Test fixes

* Add options to keep data in memory

* Fix test

* Fix lambda parameters

* Fix argument set
Fix global scope node

* Fix overload doc

* Fix stub merge errors

* Fix async issues

* Undo some changes

* Fix test

* Fix race condition

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback

* PR feedback

* Restore persistence + update test

* Better handle persistent module in dependencies

* Undo

* Add location converter abstraction

* Store member location

* Fix merge issue

* Basic locations test

* Navigation

* Add test

* Update baselines

* Type restore - initial

* Remove debug code

* Partial

* Fix stub merge

* Various model fixes

* Improve module handling

* Qualified name improvements

* Fix unbound case

* Improve stub merge

* Fix qualified names of typing

* Handle stub-only modules

* Add tests for io, re and sys

* Better handle named tuple

* Handle module circular references

* Handle else in platform and version clauses + handle it in symbol collector

* Formatting

* Fix walk of multi-level if statement

* Unify package search in imports

* Fix tests

* Undo change

* Port changes from dbtype

* Partial

* Named tuple support

* Baseline updates

* Debug code

* Support types

* Properly compare class member declaring type

* Nested classes and functions persistence

* Undo debug

* Fix numpy restore

* Baselines

* Fix tests

* Update AnyStr test reflecting changes to AnyStr behavior

* Exclude baselines from git diff

* Fix gitattr

* Move git setting to root

* Try no path

* Test fixes

* Undo change

* Additional stub merge fixes

* Baseline updates

* Fix goto def

* Protect specific type creation

* Track documentaton source

* More reliable tests + simplification

* Typo

* Cleanup

* Basic classification

* Fix merge error

* Module unique id fixes

* Stricted check to save analysis

* Revert "Fix tests"

This reverts commit 247a8c3.

* Revert "Unify package search in imports"

This reverts commit 67fed10.

* Don't clear scope variables with inner classes

* Fix typo

* Many new tests

* Fix collections test

* Fix CTypes

* Initial

* Update test

* Merge issues

* Fix CTypes again

* Fix null bases

* Tell between class members with/without self better

* TypeVar support

* Add bound/covariant

* Fix Enum reassignments

* Fix Random

* Fix import * over local declarations (Socket)

* Move interface

* Fix reference search

* Enable asyncio test

* More tests

* Enable dataclasses

* Add inspect and gzip

* More tests

* Add setting

* Add handling of import position relative to the variable

* Caching level changes

* Partial

* Rework

* Fix null

* Update baselines

* Functools pass

* Reverse stub merge

* Partial

* Partial ctypes

* Update variables post merge

* Fix ctypes

* Merge issues

* Fix typevar

* Add caching on construction

* Assorted fixes

* Named tuples

* Work around os stub oddness

* Fix generic qualified names

* Ignore lambdas

* Support named tuples as bases

* Baselines

* Test updates

* Named tuple tests

* PR feedback

* Using

* Move diagnostics to GenericBase

* PR feedback
- Fix #1455: AF: Library module of type Stub has been analyzed alread…
…y! (#1470)

- Restore task handling in unit tests
- Throw exception from Debug.Fail instead of FailFast
Synchronization
Fix import model restore
@jakebailey
Copy link
Member

left a comment

LGTM. All tests pass, and things work fine in the default state with the DB disabled.

@MikhailArkhipov MikhailArkhipov merged commit c9b1928 into master Sep 16, 2019

1 check passed

license/cla All CLA requirements met.
Details
@AlexanderSher

This comment has been minimized.

Copy link
Contributor

commented Sep 16, 2019

Let's merge it so I can apply numpy fixes over it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.