Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance when opening large projects #2271

Merged
merged 8 commits into from
Feb 8, 2022

Conversation

citizenmatt
Copy link
Member

This PR fixes a number of performance issues when opening projects that have a very large number of assets and meta files:

  • Reuse cached file metadata when creating IPsiSourceFile instances for asset and meta files. Fixes RIDER-53358
    We efficiently get this information when calling GetDirectoryEntries to discover asset and meta files in Assets, ProjectSettings and package folders, but creating the PSI source file will go back to the disk for every single file to get timestamp, length and file attributes. This has a massive impact on very large projects (tens of thousands of files), and can cause further problems on certain OS/drive/file system combos (e.g. exFAT on USB on Windows can take 12 minutes for 24,000 files just to get the information we've already got in memory)
  • Reduce excessive memory allocations in UnityExternalFilesPsiModule.SourceFiles for very large projects.
    The module's trie isn't enumerable, and would allocate and populate a new collection on each access, and it is frequently accessed. The module now maintains a separate set of files as well as the trie.
  • Reduce excessive allocations when trying to get the PSI language of UnityExternalPsiSourceFile.
    The logic to look up the PSI language would allocate, first to get file extension to look up the ProjectFileType, then allocate again in UnityYamlProjectFileLanguageService.GetPsiLanguageType when fetching the PsiLanguageType. This has been simplified to pass the ProjectFileType in to the constructor, so no more allocations and lookups are required.
    This required introducing MetaProjectFileType, which derives from YamlProjectFileType and registers the .meta file (which was previously handled by UnityYamlProjectFileType, even though meta files didn't use the UnityYaml PSI language). The appropriate project file type is calculated in the external module processor and passed in - the simple YAML project file type is used when creating files for the ProjectSettings folder.
  • Replaced a tiny allocation in a hot path in the YAML parser with a statically allocated empty NodeTypeSet. Saves several hundred megabytes on a very large project!

@citizenmatt citizenmatt added this to the Rider 2022.1 milestone Feb 8, 2022
@citizenmatt citizenmatt self-assigned this Feb 8, 2022
@citizenmatt
Copy link
Member Author

A further note on ProjectFileLanguageService, because it confused me for a while. The PsiLanguageType property is the "default" language type of the langauge service, and is only used by the language service itself. Other consumers will try to get the language type of specific files, and the base implementation will end up using the default value from PsiLanguageType.

The UnityYamlProjectFileLanguageService used to have YamlLanguage as PsiLanguageType. It would also override GetPsiLanguageType(IPsiSourceFile) and check to see if the file is .meta or ProjectSettings/*.asset. If so, it defers to base.GetPsiLanguageType(IPsiSourceFile), which ultimately returns PsiLanguageType - YamlLanguage. If it's not .meta or ProjectSettings/*.asset, then it's treated as UnityYamlLanguage.

The UnityYaml language does not have a PSI LanguageService, so won't create a lexer or parser. We manually create lexer/parser when indexing the file. This gives us control of when these potentially massive files are parsed.

UnityYamlProjectFileLanguageService has been simplified so that it only handles the UnityYaml language, and file extensions registered with UnityYamlProjectFileType - .unity, .asset, .prefab, etc. This means that if a UnityYaml file is asked for its PSI language, it will always return UnityYaml.

Asset files under ProjectSettings are now created with a YamlProjectFileType, so they will always return YamlLanguage as the PSI language.

Meta files now have MetaProjectFileType, which derives from YamlProjectFileType, so we have somewhere to register .meta, but everything else behaves like YamlProjectFileType.

@citizenmatt citizenmatt merged commit 5271f23 into net221 Feb 8, 2022
@citizenmatt citizenmatt deleted the net221-mte-project-open-perf branch February 8, 2022 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants