Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Environment info cache class #14065

Merged
merged 25 commits into from
Sep 30, 2020
Merged

Conversation

kimadeline
Copy link

@kimadeline kimadeline commented Sep 23, 2020

Note that this impacts #14020.

  • Pull request represents a single change (i.e. not fixing disparate/unrelated things in a single PR).
  • Title summarizes what is changing.
  • Has a news entry file (remember to thank yourself!).
  • Appropriate comments and documentation strings in the code.
  • Has sufficient logging.
  • Has telemetry for enhancements.
  • Unit tests & system/integration tests are added/updated.
  • Test plan is updated as appropriate.
  • package-lock.json has been regenerated by running npm install (if dependencies have changed).
  • The wiki is updated with any design decisions/details.

@kimadeline kimadeline added the no-changelog No news entry required label Sep 23, 2020
@codecov-commenter
Copy link

codecov-commenter commented Sep 23, 2020

Codecov Report

Merging #14065 into main will increase coverage by 0.00%.
The diff coverage is 67.64%.

Impacted file tree graph

@@           Coverage Diff           @@
##             main   #14065   +/-   ##
=======================================
  Coverage   59.94%   59.95%           
=======================================
  Files         705      706    +1     
  Lines       39101    39135   +34     
  Branches     5667     5674    +7     
=======================================
+ Hits        23440    23462   +22     
- Misses      14429    14438    +9     
- Partials     1232     1235    +3     
Impacted Files Coverage Δ
src/client/pythonEnvironments/index.ts 15.78% <ø> (ø)
.../pythonEnvironments/common/externalDependencies.ts 56.66% <22.22%> (-14.77%) ⬇️
src/client/pythonEnvironments/base/envsCache.ts 84.00% <84.00%> (ø)
src/client/common/utils/platform.ts 68.00% <0.00%> (-4.00%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6332b81...d67a118. Read the comment docs.

@kimadeline kimadeline marked this pull request as ready for review September 23, 2020 22:44
Copy link

@karrtikr karrtikr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some changes can depend on the resolution of https://github.com/microsoft/vscode-python/pull/14020/files#r493894442

src/client/pythonEnvironments/base/envsCache.ts Outdated Show resolved Hide resolved
src/client/pythonEnvironments/base/envsCache.ts Outdated Show resolved Hide resolved
Co-authored-by: Kartik Raj <karraj@microsoft.com>
Copy link
Member

@ericsnowcurrently ericsnowcurrently left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good. I've left a variety of comments, some potentially related to correctness. I'd be glad to discuss any of it.

src/client/pythonEnvironments/index.ts Outdated Show resolved Hide resolved
src/client/pythonEnvironments/base/envsCache.ts Outdated Show resolved Hide resolved
src/client/pythonEnvironments/base/envsCache.ts Outdated Show resolved Hide resolved
src/client/pythonEnvironments/base/envsCache.ts Outdated Show resolved Hide resolved
src/test/pythonEnvironments/base/envsCache.unit.test.ts Outdated Show resolved Hide resolved
src/client/pythonEnvironments/base/envsCache.ts Outdated Show resolved Hide resolved
src/client/pythonEnvironments/base/envsCache.ts Outdated Show resolved Hide resolved
@kimadeline
Copy link
Author

No idea why some of the PR checks are still failing 🤷‍♀️ Not related to this PR since I only added unit tests.

Copy link
Member

@ericsnowcurrently ericsnowcurrently left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks for being so diligent about addressing the review comments!

Copy link

@karrtikr karrtikr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -120,7 +118,6 @@ export function registerTypes(serviceManager: IServiceManager) {
serviceManager.addSingleton<IInterpreterPathService>(IInterpreterPathService, InterpreterPathService);
serviceManager.addSingleton<IExtensions>(IExtensions, Extensions);
serviceManager.addSingleton<IRandom>(IRandom, Random);
serviceManager.addSingleton<IPersistentStateFactory>(IPersistentStateFactory, PersistentStateFactory);
serviceManager.addSingleton<ITerminalServiceFactory>(ITerminalServiceFactory, TerminalServiceFactory);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IPersistentStateFactory is not specific to python environments, why're we moving it?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When calling cache.initialize() inside the python environments initialization it needs access to the persistent state factory, however the latter was being added to the service manager later in the process, causing the entire extension initialization to fail.

Copy link
Author

@kimadeline kimadeline Sep 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like it is causing all the other tests to fail though.

An alternative could be to not initialize the cache when calling the python envs initialization method, and let the cache locator deal with that.

Another alternative could be to add a "post-activate" call after activateComponents, where stuff like the envs cache would be initialized.

/cc @ericsnowcurrently

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

however the latter was being added to the service manager later in the process

I think that should be changed. We need to ensure that we do all the registrations before initializing the python envs component. Basically time to do the TODO item,

// tslint:disable-next-line:no-suspicious-comment
// TODO: Gradually move simple initialization
// and DI registration currently in this function over
// to initializeComponents(). Likewise with complex
// init and activation: move them to activateComponents().
// See https://github.com/microsoft/vscode-python/issues/10454.

Put them all before line 43 here,

export function initializeComponents(
_context: IExtensionContext,
serviceManager: IServiceManager,
serviceContainer: IServiceContainer
) {
activatePythonEnvironments(serviceManager, serviceContainer);
// We will be pulling code over from activateLegacy().
}

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is going to touch a couple of moving parts so to wrap up this PR I will revert the last activation changes I made and not initialize the cache. I'll set up a separate PR to move activation of what the cache needs to initializeComponents, and re-add cache initialization.

@sonarcloud
Copy link

sonarcloud bot commented Sep 30, 2020

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities (and Security Hotspot 0 Security Hotspots to review)
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@kimadeline
Copy link
Author

OMG

image

Using up all my good karma for the day and merging it 💨

@kimadeline kimadeline merged commit 81b6d8f into microsoft:main Sep 30, 2020
@kimadeline kimadeline deleted the environments-cache branch September 30, 2020 17:10
@kimadeline kimadeline mentioned this pull request Sep 30, 2020
10 tasks
rchiodo added a commit that referenced this pull request Oct 1, 2020
* Add an optional onUpdated event to the iterator returned by ILocator.iterEnvs(). (#13950)

In order to ensure that ILocator.iterEnvs() can finish as fast as possible, we add a side-channel event for each iteration. This event fires any time an already-yielded env info object is updated (e.g. reduced/merged or resolved/completed). The update only relates to operations triggered by that particular iteration. ILocator.onChanged remains separate and only relates to when the locators finds a new env, notices one was removed, or that one was otherwise fundamentally changed.

There are two similar approaches we could take for this update event. Either we added the event as a property of IPythonEnvsIterator or we change the return value of ILocator.iterEnvs() to be a 2-tuple ([PythonEnvsIterator, Event<PythonEnvUpdatedEvent>]). We took the property approach since most of the time callers won't need to worry about the update events.

* fully bump raw kernel (#13941)

Co-authored-by: Ian Huff <ianhuff@CEIDCCEVIPSVC01.redmond.corp.microsoft.com>

* Virtualenvwrapper locator (#13895)

* virtualenvwrapper locator

* skip os-specific tests

* Test getDefaultVirtualenvwrapperDir

* Update src/client/pythonEnvironments/discovery/locators/services/virtualenvwrapperLocator.ts

Co-authored-by: Karthik Nadig <kanadig@microsoft.com>

* add pathExists check to WORKON_HOME dir

* Change how the locator works + move util func out

* Fix functional tests

* utils unit tests

* Update locator tests

* Stub getUserHomeDir instead of getDefaultetc

Co-authored-by: Karthik Nadig <kanadig@microsoft.com>

* Clean up eslint errors (#13951)

* Clean up eslint errors

* Fix linting and address comments.

* Add a legacy DI adapter for the Python envs component. (#13858)

This allows us to start using the new discovery code in the extension.

* Extension api for DataScience (#13923)

* Extension API
* Api for DS

* Fix installing ipykernel into interpreters for raw kernel (#13959)

* Fix kernel and server name missing in certain situations (#13974)

* Fix kernel name and server name

* Fixup server name for remote situations

* Add some functional tests

* Add news entry

* Extension API to return path to debugpy for DS (#13973)

* Use the component adapter in the extension. (#13869)

This allows us to start using the new discovery code in the extension. The key thing is to be careful not to regress in the available functionality. So for now we have disabled use of the component.  When we feel comfortable with it we can enable it by setting the default for `ComponentAdapter.enabled` to `true`.

This PR involves small fixes to a large number of files due to API changes. The adapter is actually used (injected) in the following files:

src/client/interpreter/interpreterService.ts
src/client/interpreter/helpers.ts
src/client/pythonEnvironments/discovery/locators/index.ts
src/client/pythonEnvironments/discovery/locators/services/condaService.ts
src/client/pythonEnvironments/discovery/locators/services/windowsStoreInterpreter.ts

* Disable split views of custom editors (#13985)

* Fix backup storage by looking at the options correctly (#13983)

* Fix backup storage by looking at the options correctly

* Fix backup by being more explicit

* Only linux tests are failing. Hopefully fix them

* update test plan for ipykernel (python repo) (#13993)

* Find interpreter based on hash in kernelspec of nb metadata (#13856)

* Pyenv locator (#13996)

* Pyenv locator
* Skip tests per platform
* Wrong pyenv path order in ternary
* Add description
* Autoformat venv locator
* Revert "Autoformat venv locator"
This reverts commit 5c8c4ab.
* Add links
* Windows-specific fixes
* Typo

* merge

* Revert "merge"

This reverts commit ee0cad9.

* Remove cwd while running using isolate script (#14014)

* Remove cwd while running using isolate script

* Run inplace.

* Fix formatting

* Upgrade isort to 5.5.3 (#14035)

* Modify environment info worker to support new type and to work with resolver (#13997)

* Modify environment info worker to support new environment type

* Do not replace existing fields and return a new object

* Modify cache to carry deferred instead

* Change worker to return interpreter information instead

* Handle error and rename

* Fix bug with interpreterInfo.py

* Code reviews

* Rename old PythonEnvInfo type to InterpreterInfoJson

* Remove cross process lock (#14036)

* Add API to get language server from external extensions (#14021)

* Ideas for lang server API with python extension

* Working with new API

* Minimize API surface

* Fix up tests for intellisense

* Fix unit tests

* Put back custom editor service

* Remove unnecessary changes for service registry

* Code review feedback

* Move connection type to types file

* Fix failing virtualenvwrapper Windows unit tests (#14012)

* Fix tests
* Typo
Co-authored-by: Karthik Nadig <kanadig@microsoft.com>
Co-authored-by: Karthik Nadig <kanadig@microsoft.com>

* make sure that we run our show code when a custom editor is shown (#14022)

* make sure that we run our show code when a custom editor is shown

* Trigger notebook load in the constructor, not show

Co-authored-by: Ian Huff <ianhuff@CEIDCCEVIPSVC01.redmond.corp.microsoft.com>

* Rely on AppInsights API for exceptions (#13878)

* Fix path

* Actually fix settings

* Add news

* Add test

* Format

* Suppress 'jediEnabled' removal

* Drop survey first launch threshold

* Telemetry for exceptions

* Remove custom exception telemetry

* Remove unused

* Fix HTML escaping to match what Jupyter does (#14038)

* Basic idea

* add some functional tests

* Add news entry

* Fix functional tests

* Have dependabot ignore pytest and py (#14039)

Otherwise it doesn't understand the Python 2.7 restrictions for those dependencies.

* Added environments reducer (#13953)

* Add environments reducer

* Added tests

* Use path.join to construct paths

* Code reviews

* Correct dummy implementations and adjust tests

* Modify resolveEnv()

* Rename to a general parentLocator

* Merge release back to master (#14059)

* Port ipykernel install fix to release (#13975)

* Fix installing ipykernel into interpreters for raw kernel (#13959)

* update news

Co-authored-by: Ian Huff <ianhuff@CEIDCCEVIPSVC01.redmond.corp.microsoft.com>

* Merge in changes to release (#13976)

* Up release version for new release (#13928)

* Up release version

* Update changelog

* Update changelog

* Workaround test issue (#13930)

* Try different version of VS code in release

* Change to make it use the actual variable

* Use a real version

* More tests failing with gpu error (#13935)

* Try different version of VS code in release

* Change to make it use the actual variable

* Use a real version

* Two more version changes

* Fix kernel and server name missing in certain situations (#13974)

* Fix kernel name and server name

* Fixup server name for remote situations

* Add some functional tests

* Add news entry

* Delete news file

* Port two fixes to the release branch (#13995)

* Disable split views of custom editors (#13985)

* Fix backup storage by looking at the options correctly (#13983)

* Fix backup storage by looking at the options correctly

* Fix backup by being more explicit

* Only linux tests are failing. Hopefully fix them

* Fixup changelog

Co-authored-by: Don Jayamanne <don.jayamanne@yahoo.com>

* add jedi-language-server to 3rd party notices (#13977)

* add jedi-language-server to 3rd party notices

* move license from distribution to repository file

* disable test_discover_complex_default and (#14024)

test_discover_complex_doctest

* Upgrade isort to 5.5.3 (#14035) (#14037)

* prepare release (#14042)

* fixed annoying warnings (#14049)

* update version

Co-authored-by: Ian Huff <ianhu@microsoft.com>
Co-authored-by: Ian Huff <ianhuff@CEIDCCEVIPSVC01.redmond.corp.microsoft.com>
Co-authored-by: Rich Chiodo <rchiodo@users.noreply.github.com>
Co-authored-by: Don Jayamanne <don.jayamanne@yahoo.com>
Co-authored-by: Kartik Raj <karraj@microsoft.com>

* Fix object sort order in tools tests (#14050)

* Fix object sort order in tools tests

* rebase with main

* Fix formatting

* Update ubuntu images for GPU issue fix (#14064)

* Update ubuntu images for GPU issue fix

* Try with ubuntu-20.04

* Added environments resolver (#14019)

* Modify environment info worker to support new environment type

* Change worker to return interpreter information instead

* Modify resolveEnv()

* Code reviews

* Code reviews

* Move stuff

* Add Windows unit tests to the PR pipeline (#14106)

* Do not opt users out of the insiders program if they have a stable version installed (#14091)

* Remove code that opts users out of the insiders program

* News entry

* Code reviews

* Do not quote isolated in exec module (#14108)

* Do not quote isolated in exec module

* Revert "Do not quote isolated in exec module"

This reverts commit b9fa04c.

* Revert "IPyKernel install issue with windows paths (#13667)"

This reverts commit 23725ab.

* Update cell output and metadata using Edit API (#13737)

* Fix unit test broken by recent revert (#14122)

* revert VSC_PYTHON_CI_TEST_VSC_CHANNEL changes (#14109)

* revert VSC_PYTHON_CI_TEST_VSC_CHANNEL changes

* Revert "revert VSC_PYTHON_CI_TEST_VSC_CHANNEL changes"

This reverts commit 6ed8f61.

* Remove force 1.48 on Test VSC channel

* Merge environment and compare environments (#14026)

* Add some heuristic functions

* Support merging.

* Clean up identifier code

* Tweaks and fixes

* More clean up.

* versions tweak

* Address comments

* more comments

* Fix merge issues.

* Fix more merge issues.

* Add PythonEnvInfo-related helpers. (#14051)

This PR adds some basic helpers that we use in a subsequent PR. The following small drive-by changes are also included:

* drop PythonEnvInfo.id property
* make some internal helpers public

* Escaping fix broke a number of things (#14145)

* Fixes for escaping

* Push a comment ot start PR again

* Cache task is failing

* Remove cache task

* Not fixing so just put back cache task

* Auto assign datascience issues (#14147)

* Change name of event used to activate extension (#14153)

* Fixes to streamed output in native notebooks (#14158)

For #13611
* Fixes to streamed output
* added tests

* Fixes to for escaping of output in native notebooks (#14159)

* Look for `data science` labels when auto assigning issues (#14161)

* Environment info cache class (#14065)

* Add persistent storage external deps
* PythonEnvInfoCache class + tests
* Instantiate & initialize cache class in createAPI
* Add extra test for flush() and initialize()
* Env cache fixes: storage key + find() result check
* Update src/client/pythonEnvironments/common/externalDependencies.ts
Co-authored-by: Kartik Raj <karraj@microsoft.com>
* Use areSameEnvironment in getEnv
* Don't ping persistent storage for every initialize
* No need to export CompleteEnvInfoFunction
* PythonEnvInfoCache doc comment
* Rename createGlobalPersistentStoreStub to get...
* Preemptively drop id key (#14051)
* Return deep copies
* IPersistentStore wrapper around IPersistentState
* Use correct areSameEnvironment + fix stub
* Remove obsolete comment
* getEnv -> filterEnvs
* Remove stubbing of areSameEnvironment
* Update areSameEnv
* Move IPersistentStateFactory registration to registerForIOC
* Revert "Move IPersistentStateFactory registration to registerForIOC"
This reverts commit edc6ce5.
* Don't instantiate nor initialize cache for now
Co-authored-by: Kartik Raj <karraj@microsoft.com>

* Update Component Governance notes in the release plan

* Merge point release back into main (#14171)

* Port ipykernel install fix to release (#13975)

* Fix installing ipykernel into interpreters for raw kernel (#13959)

* update news

Co-authored-by: Ian Huff <ianhuff@CEIDCCEVIPSVC01.redmond.corp.microsoft.com>

* Merge in changes to release (#13976)

* Up release version for new release (#13928)

* Up release version

* Update changelog

* Update changelog

* Workaround test issue (#13930)

* Try different version of VS code in release

* Change to make it use the actual variable

* Use a real version

* More tests failing with gpu error (#13935)

* Try different version of VS code in release

* Change to make it use the actual variable

* Use a real version

* Two more version changes

* Fix kernel and server name missing in certain situations (#13974)

* Fix kernel name and server name

* Fixup server name for remote situations

* Add some functional tests

* Add news entry

* Delete news file

* Port two fixes to the release branch (#13995)

* Disable split views of custom editors (#13985)

* Fix backup storage by looking at the options correctly (#13983)

* Fix backup storage by looking at the options correctly

* Fix backup by being more explicit

* Only linux tests are failing. Hopefully fix them

* Fixup changelog

Co-authored-by: Don Jayamanne <don.jayamanne@yahoo.com>

* add jedi-language-server to 3rd party notices (#13977)

* add jedi-language-server to 3rd party notices

* move license from distribution to repository file

* disable test_discover_complex_default and (#14024)

test_discover_complex_doctest

* Upgrade isort to 5.5.3 (#14035) (#14037)

* prepare release (#14042)

* fixed annoying warnings (#14049)

* Cherry pick to address path issues. (#14125)

* Do not quote isolated in exec module (#14108)

* Do not quote isolated in exec module

* Revert "Do not quote isolated in exec module"

This reverts commit b9fa04c.

* Revert "IPyKernel install issue with windows paths (#13667)"

This reverts commit 23725ab.

* Fix unit test broken by recent revert (#14122)

* Port escape fix to release branch (#14133)

* Fix HTML escaping to match what Jupyter does (#14038)

* Basic idea

* add some functional tests

* Add news entry

* Fix functional tests

* Update changelog

* update version and changelog (#14139)

* Escaping fix broke a number of things (#14145) (#14154)

* Fixes for escaping

* Push a comment ot start PR again

* Cache task is failing

* Remove cache task

* Not fixing so just put back cache task

* remove leftovers

* import pytest

* stop skipping python tests

Co-authored-by: Ian Huff <ianhu@microsoft.com>
Co-authored-by: Ian Huff <ianhuff@CEIDCCEVIPSVC01.redmond.corp.microsoft.com>
Co-authored-by: Rich Chiodo <rchiodo@users.noreply.github.com>
Co-authored-by: Don Jayamanne <don.jayamanne@yahoo.com>
Co-authored-by: Kartik Raj <karraj@microsoft.com>
Co-authored-by: Karthik Nadig <kanadig@microsoft.com>

* NB Convert 6.0 support for export (#14177)

* Windows store locator (#14162)

* Initial commit for windows store locator

* More tests

* Simplify locator

* Tweaks

* Test fixes

* Fix tests

* A different way of fixing escaping (#14186)

* Move escaping to just output

* Add some tests to verify round tripping

* Fixup test for rountrip and make roundtripping actually work

* Add news entry

* Add to manual test file

* Fix streaming problem and add more to the test

* Fix traceback unit test

* Fix problem caught by functional tests :)

* Another functional test catch

* Fixes to escaping of output in native notebooks (#14199)

* Display description in kernel picker (#14196)

* Bump node-fetch to 2.6.1 (#14181)

* Bump pytest from 6.0.1 to 6.1.0 in /news (#14190)

Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.0.1 to 6.1.0.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@6.0.1...6.1.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Messed up merge commits

Co-authored-by: Eric Snow <eric.snow@microsoft.com>
Co-authored-by: Ian Huff <ianhu@microsoft.com>
Co-authored-by: Ian Huff <ianhuff@CEIDCCEVIPSVC01.redmond.corp.microsoft.com>
Co-authored-by: Kim-Adeline Miguel <51720070+kimadeline@users.noreply.github.com>
Co-authored-by: Karthik Nadig <kanadig@microsoft.com>
Co-authored-by: Don Jayamanne <don.jayamanne@yahoo.com>
Co-authored-by: Graham Wheeler <gram@geekraver.com>
Co-authored-by: Kartik Raj <karraj@microsoft.com>
Co-authored-by: Joyce Er <joyceerhuiling@gmail.com>
Co-authored-by: Mikhail Arkhipov <mikhaila@microsoft.com>
Co-authored-by: Brett Cannon <brett@python.org>
Co-authored-by: David Kutugata <dakutuga@microsoft.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
no-changelog No news entry required
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants