Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf: asynchronously retrieve all collection info (schematics, generators, builders and executors) #1160

Merged
merged 7 commits into from
Oct 22, 2021
Merged

perf: asynchronously retrieve all collection info (schematics, generators, builders and executors) #1160

merged 7 commits into from
Oct 22, 2021

Conversation

Cammisuli
Copy link
Member

@Cammisuli Cammisuli commented Oct 16, 2021 •
edited

This PR changes the way that Nx Console retrieves all collection info (schematics, generators, builders and executors).

Previous Implementation

Before, file reads from node_modules were done synchronously. This obviously caused some high CPU load on slower machines. This has also been the cause of the extension host crashing recently

New implementation

With this change, Nx Console will now only look into the dependencies asynchronously.

This also combines the builder(/executor) and schematics (/ generators) lookup so that we don't have to constantly look into the node_modules for this information.

Fixes #1155
Fixes #1153

@nx-cloud
Copy link

nx-cloud bot commented Oct 16, 2021 •
edited

Nx Cloud Report

CI ran the following commands for commit 42c5e57. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this branch

Status Command
#000000 nx run-many --all --parallel --target=build
#000000 nx run-many --all --parallel --target=test
#000000 nx run-many --all --parallel --target=lint

Sent with 💌 from NxCloud.

@Cammisuli Cammisuli changed the title perf: asynchronously retrieve all collection info (schematics, generators, builders and executors) WIP perf: asynchronously retrieve all collection info (schematics, generators, builders and executors) Oct 16, 2021
@Bullfrog1234
Copy link

If you are not sure this is breaking change could it be created so that a flag is added to the settings that disables the search through node_modules allowing those with issues to go and disable it if they want, and those without to not.

Just an idea.

@Cammisuli
Copy link
Member Author

I don't know if I'd like reading through the whole node_modules. That always seemed kind of overkill. I always wanted to streamline this to only read the dependencies that are actually listed in the package.json.

Adding a flag to keep the old behaviour doesn't feel right for me.

And the reason why this could potentially be a breaking change is if someone was using schematics/generators that they didn't explicitly add to their dependencies. To me, that's not the correct way and those dependencies should be listed.

I don't know yet, I'd like to gather some more opinions and see what people think.

Thanks for your feedback 😄

@Cammisuli Cammisuli changed the title WIP perf: asynchronously retrieve all collection info (schematics, generators, builders and executors) perf: asynchronously retrieve all collection info (schematics, generators, builders and executors) Oct 22, 2021
@Cammisuli Cammisuli merged commit e193eb8 into nrwl:master Oct 22, 2021
@Cammisuli Cammisuli deleted the async-collection-retrieval branch October 22, 2021 20:57
@Bullfrog1234
Copy link

I don't know if I'd like reading through the whole node_modules. That always seemed kind of overkill. I always wanted to streamline this to only read the dependencies that are actually listed in the package.json.

Adding a flag to keep the old behaviour doesn't feel right for me.

And the reason why this could potentially be a breaking change is if someone was using schematics/generators that they didn't explicitly add to their dependencies. To me, that's not the correct way and those dependencies should be listed.

I don't know yet, I'd like to gather some more opinions and see what people think.

Thanks for your feedback 😄

Was only a suggestion to how to have an opt-in non-breaking change approach, but in no way saying had to take it.

I agree with you on the dependencies should be listed. And that we should work away from scanning the node-module.

Cammisuli pushed a commit that referenced this pull request Aug 9, 2022
@semantic-release-bot
chore(release): 1.0.0 [skip ci]
48c84ec 
# 1.0.0 (2022-08-09)

### Bug Fixes

* add authentication boolean to settings ([a2a1a3f](a2a1a3f))
* add dotenv as a dep ([691cc7a](691cc7a))
* artifact name on windows ([eeb5404](eeb5404))
* await workspace reset ([2e9d880](2e9d880))
* build ([#754](#754)) ([fc87f6b](fc87f6b))
* **build:** add EV certificate and condition to clean script ([0d8bac5](0d8bac5))
* **build:** add missing deps ([61d73c3](61d73c3))
* builders should show the values from angular.json ([8aa7c0a](8aa7c0a))
* **build:** fix linux artifact name ([16b9660](16b9660))
* **build:** fix windows build ([a1df664](a1df664))
* **build:** Update graphql types ([#872](#872)) ([91fbe86](91fbe86))
* bump up electron version to fix security vulnerability ([a1c3f6f](a1c3f6f))
* Bundle visualization accounts for differential loading ([2bc8ae0](2bc8ae0))
* check if dependency path type is directory with bit flag ([#1267](#1267)) ([e75daac](e75daac))
* command status should return an empty array ([b3b0211](b3b0211))
* **deps:** update dependency @angular/cli to v7.0.6 ([2f62cae](2f62cae))
* **deps:** update dependency apollo-server-express to v2.2.2 ([b680263](b680263))
* **deps:** update dependency drivelist to v6.4.3 ([87cead5](87cead5))
* **deps:** update dependency electron-updater to v3.2.3 ([5373827](5373827))
* **deps:** update dependency electron-updater to v4 ([#383](#383)) ([2d8b045](2d8b045))
* **deps:** update dependency express-graphql to v0.7.1 ([e04044c](e04044c))
* **deps:** update dependency get-port to v4 ([#381](#381)) ([e8acf5c](e8acf5c))
* **deps:** update dependency node-pty-prebuilt to v0.7.6 ([e4ae6eb](e4ae6eb))
* **deps:** update dependency universal-analytics to v0.4.20 ([b2e52c7](b2e52c7))
* **deps:** update nrwl monorepo to v0.0.17 ([573d687](573d687))
* Do not prompt for analytics ([#757](#757)) ([aca0c4a](aca0c4a))
* **electron:** icon on a mac ([feb99b1](feb99b1))
* **electron:** Use correct cwd for ng-new ([#500](#500)) ([3897b71](3897b71))
* **error-handling:** Give more feedback for common errors ([#881](#881)) ([e343fbd](e343fbd))
* fix auto updater ([5c52096](5c52096))
* Fixed error output of build task. ([#563](#563)) ([827b016](827b016))
* Fixed scripts so that `yarn startdev.prepare` work ([425f8fa](425f8fa))
* handle hyphenation when looking for generator defaults ([#1315](#1315)) ([60bdadf](60bdadf)), closes [#1313](#1313)
* **icons:** Fix icons on linux ([#430](#430)) ([4b1ba6a](4b1ba6a))
* improve performance for retrieving workspace details ([#1295](#1295)) ([f8d32ca](f8d32ca))
* inline copy-to-clipboard icon ([#1220](#1220)) ([5e113df](5e113df))
* **many:** Various small bug fixes ([#700](#700)) ([d0eb2e9](d0eb2e9))
* misc fixes ([b577ca9](b577ca9))
* misc fixes ([bc6cb95](bc6cb95))
* **new-workspace:** Create a sandbox to create new workpaces ([be0722d](be0722d))
* node pty version for vscode ([#524](#524)) ([868b96c](868b96c))
* not all schematic collections get picked up and displayed ([1b98dd9](1b98dd9))
* only reads and normalizes selected schematic options ([#1074](#1074)) ([d2c4bd4](d2c4bd4))
* project creation works with angular 8 ([b7e9818](b7e9818))
* **projects:** Disable generate/run buttons when applicable ([#763](#763)) ([9b7ea68](9b7ea68))
* properly check positional argument for generating with UI ([#1268](#1268)) ([764b95f](764b95f))
* Reduce logging noise and handle schematics without a description ([#750](#750)) ([3d15f03](3d15f03))
* revert electron-builder version to work with electron 2 ([cc2f336](cc2f336))
* run executors that are local to the workspace (ie local executors) ([#1230](#1230)) ([a7f2362](a7f2362))
* run pnp.setup to patch require ([#1240](#1240)) ([919e05d](919e05d))
* scripts section gets removed from package.json ([ea65ec7](ea65ec7))
* **settings:** Set initial value of disabled animations ([#657](#657)) ([f11f2d5](f11f2d5))
* tweak polling constants to improve perf ([#459](#459)) ([93a62ed](93a62ed))
* validation on workspace name to prevent conflicts ([35765c8](35765c8))
* **vscode-dev:** Copy server assets as part of ng build ([#802](#802)) ([fa208e9](fa208e9))
* **vscode:** Make sure that karma.conf.js is in assets folder ([#482](#482)) ([e601c7a](e601c7a))
* **vscode:** More lenient JSON parsing (allow for trailing commas) ([#902](#902)) ([ecbd0ee](ecbd0ee))
* **vscode:** Remove all logging ([#758](#758)) ([3fd4ffc](3fd4ffc))
* **windows:** strip out node-pty-prebuilt for vscode and intellij ([#745](#745)) ([cb104ce](cb104ce))

### Features

* Add commands to add applications and libraries directly ([#1128](#1128)) ([b908c11](b908c11))
* add configuration support for the import plugin ([#1183](#1183)) ([53b09eb](53b09eb))
* add detailed status for build and serve ([765e06e](765e06e))
* add detailed status for commands ([3a0e90c](3a0e90c))
* add different release channels ([21159d8](21159d8))
* add google analytics ([72dbaea](72dbaea))
* add openInBrowser functionality to be used by plugins ([2f2ad75](2f2ad75))
* add prompts to add nx to angular cli ([#1254](#1254)) ([a669307](a669307))
* add support for docs ([0bfae75](0bfae75))
* add support for notifications ([0d5a78f](0d5a78f))
* add support for workspace schematics ([dcaf8a2](dcaf8a2))
* add support for yarn pnp ([#1213](#1213)) ([b6cfdbf](b6cfdbf))
* adds settings page and fixes tests output ([#313](#313)) ([58c7387](58c7387))
* **build:** Add Renovate to automatically create PRs for updating packages ([62de102](62de102))
* **build:** Bring vscode and electron builds more inline ([#481](#481)) ([7cd4fcc](7cd4fcc))
* **build:** Build for linux distros ([3c3abea](3c3abea))
* **build:** Build images for linux ([b5ce360](b5ce360))
* **build:** New GraphQL Code Generator ([#413](#413)) ([6677531](6677531))
* **build:** Reduce external dependencies ([#622](#622)) ([c5a2a52](c5a2a52))
* **build:** Smaller bundles using esnext and minimal assets ([#747](#747)) ([c1fb0e1](c1fb0e1))
* **build:** Update electron / render linux icons ([53c1448](53c1448))
* **build:** Use GraphQL Code Generator and Apollo-Angular template ([6e83dcb](6e83dcb))
* bump up version of extensions ([22243ea](22243ea))
* bump up version of extensions ([b943ebe](b943ebe))
* **ci:** Cache yarn dependencies and default to use yarn for ng commands. ([28402e2](28402e2))
* **ci:** No longer persist workspace ([#548](#548)) ([8a7d33d](8a7d33d))
* **ci:** Run CI on windows too ([#769](#769)) ([dc4a3ca](dc4a3ca))
* dep-graph integration ([4e4d26d](4e4d26d))
* **e2e:** Speed up prepare.e2e ([#544](#544)) ([eb6a51f](eb6a51f))
* **electron:** Build windows portable installer ([#805](#805)) ([619b1a8](619b1a8))
* enable connect plugin ([8341d30](8341d30))
* **formatting:** Update prettier to 1.15 ([#322](#322)) ([c36a5e1](c36a5e1))
* **IDE:** Initial push for webstorm plugin ([#607](#607)) ([503f573](503f573))
* implement concurrent commands ([ab9630f](ab9630f))
* include typescript plugin for imports ([#1177](#1177)) ([9c330ce](9c330ce))
* **lint:** Add style lint for scss linting ([#631](#631)) ([82ceef9](82ceef9))
* list local plugins ([#1274](#1274)) ([2e6fe76](2e6fe76))
* **ng-new:** Allow users to set ng new schematic flags. ([#328](#328)) ([cf1bb36](cf1bb36))
* **node:** Add api endpoints to detect/install node ([756341e](756341e))
* **nvm:** Add support for .nvmrc files ([#583](#583)) ([6c3e975](6c3e975))
* **nvm:** Add toggle to opt-in/out of nvm ([#601](#601)) ([b05e53e](b05e53e))
* nx lsp ([#1316](#1316)) ([ab8031c](ab8031c))
* **output:** Adds visualization for serve and build output. ([#557](#557)) ([672b5ef](672b5ef))
* **output:** Adds webpack analyzer ([#573](#573)) ([786585a](786585a))
* pinned projects ([#478](#478)) ([a47c438](a47c438))
* project graph integration ([#1305](#1305)) ([ebcd217](ebcd217))
* provide json schema for workspace.json ([#1077](#1077)) ([ae35ec5](ae35ec5))
* provide telemetry to console plugins ([a5049b4](a5049b4))
* Rework analytics ([#801](#801)) ([795bd63](795bd63))
* **schema:** Remove divergent schemas in favor of common ones ([#537](#537)) ([caf4ebf](caf4ebf))
* separate the history of commands from "recent" ([7a5524a](7a5524a))
* **server:** Use angular-cli's built in parser ([#913](#913)) ([c43c7a1](c43c7a1))
* **settings:** Toggle to disable animations completely ([#611](#611)) ([56afb95](56afb95))
* Show all actions from project view ([#490](#490)) ([5e09c0e](5e09c0e))
* show generate ui from explorer context menu ([8b45841](8b45841))
* shows argument list and highlights changed and invalid fields in menu ([#1059](#1059)) ([7d8e8e1](7d8e8e1))
* support nx.json extends property ([#1124](#1124)) ([507ce24](507ce24))
* **support:** Initial commit of angular-console-nrwl-extensions integration ([6b62165](6b62165))
* **tasks:** Add UI for background tasks ([b4c3293](b4c3293))
* **tasks:** Kill tasks via ctrl-c ([#327](#327)) ([1d41c69](1d41c69))
* update angular-apollo ([588274d](588274d))
* update extensions ([27b5e1d](27b5e1d))
* update nx to 6.2.1 ([f3e29a8](f3e29a8))
* update nx to 7.3.0 ([9e72d5c](9e72d5c))
* update the depedency diagram ([a0f8106](a0f8106))
* update to cli7 and nx7 ([#312](#312)) ([d6b16e1](d6b16e1))
* update to nx 7.0.1 ([a7bce06](a7bce06))
* update to nx8 and cli8 ([280ea39](280ea39))
* use x-prompt details in nx-console-field ([7537387](7537387))
* Uses source map to display graph data and fixes Windows path issue. ([#585](#585)) ([d80aca4](d80aca4))
* **vscode:** Angular JSON tree provider ([#810](#810)) ([31ceb9d](31ceb9d))
* **vscode:** Expose tree view ([0484fa1](0484fa1))
* **vscode:** Initial release of VS Code plugin ([a250473](a250473))
* **vscode:** Integrate with vscode-ui ([#839](#839)) ([6c02464](6c02464))
* **vscode:** Reuse terminals for other tasks ([#783](#783)) ([9861fe6](9861fe6))
* **vscode:** Support angular workspaces in subdirectories ([#687](#687)) ([2a64c28](2a64c28))
* **vscode:** Support workspace.json for non Angular workspaces ([#926](#926)) ([b3954c0](b3954c0))
* **vscode:** Use vscode tasks ([#792](#792)) ([2245b10](2245b10))
* **vscode:** vscode-ui form inputs ([#840](#840)) ([9c9e75e](9c9e75e))

### Performance Improvements

* asynchronously retrieve all collection info (schematics, generators, builders and executors) ([#1160](#1160)) ([e193eb8](e193eb8))
* decrease activation time ([#1131](#1131)) ([6a23cea](6a23cea))
* increase performance of nx console ([#1038](#1038)) ([e8f3258](e8f3258))
* No longer poll for updates ([#690](#690)) ([198c51e](198c51e))

### Reverts

* Revert "chore(deps): update dependency @angular-devkit/core to v7 (#372)" (#389) ([569cf2b](569cf2b)), closes [#372](#372) [#389](#389)
* Revert "fix: add validation when creating new workspace with conflicting name" ([34e9ac5](34e9ac5))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

nx console - Extension host terminated unexpectedly. Extension causes high cpu load
2 participants
@Cammisuli @Bullfrog1234