Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

As a smith I want to parallelize the builder of lsp #1552

Closed
mmews-n4 opened this issue Nov 18, 2019 · 1 comment
Closed

As a smith I want to parallelize the builder of lsp #1552

mmews-n4 opened this issue Nov 18, 2019 · 1 comment
Assignees

Comments

@mmews-n4
Copy link
Contributor

Currently, the lsp builder processes every workspace project sequentially. This could be improved by first creating a dependency graph and process projects in parallel that are independent from each other.

@mmews-n4 mmews-n4 self-assigned this Nov 18, 2019
mmews-n4 added a commit that referenced this issue Nov 18, 2019
mmews-n4 added a commit that referenced this issue Nov 27, 2019
…er (Part 4) (#1547)

* fix double read of resource in case of issues

* integrated some N4JS subclasses into xtext copies

* some fixes: NPEs, wrong projects, JavaDoc

* add validation for wrong argument

* add validation for wrong argument and test

* enable restart of LSP server

* fix NPE when adding a FQN

* merge (copy/paste) some changes from GH-1538

* add isIndexOnly to IProjectConfig to be checked when creating in Builder

* add isIndexOnly to IProjectConfig to be checked when creating in Builder

* fix test helpers to cope with building libs from mvn call

* lots of refactorings to merge N4JS- and X-classes, and to reduce lambdas

* refactored BuildRequest, introduced IssueAcceptor

* introduce ProjectStateHolder to encapsule project state

* refactor (move) project state persister

* store and load validation issues to/from disk

* switch to new compiler for n4js-libs

* NPE fix

* fix wrong return values

* add initializers to Issues

* propagate exceptions during validation

* fix sorting of Non-comparables (was never working)

* support clean and non-persisting builds

* refactor XIncrementalBuilder: extract inner classes

* add switch to clean before compile or lsp

* add option --noPersist

* clean-up of changes due to github.com/eclipse/xtext-core/issues/1209

* fix missing package.json in the source containers

* fix test and align doc of CliCompileResult

* fix wrong path

* move up assignment since resourceSet is used already during build

* clean-ups: remove obsolete code

* enable shadowing until GH-1314 is done

* fix project discovery for yarn main node_modules folder and npms

* fix getAllResources to exclude files from node_modules folders

* fix performance issue: introduce caching for external libraries

* refactoring to reduce lambdas

* set as external when inside of node_modules folder

* set to warning due to GH-1314

* add experimental GH-1552 (inactive)

* replace the old n4jsc.jar by the new one

* fix tests due to switch from error to warning

* adjust n4js-libs to new n4jsc.jar

* adjust help text of n4jsc.jar

* remove obsolete "--clean" call to n4jsc.jar

* comment TODO

* support writing test catalog in new n4jsc

* fix tag regarding semver syntax

* support cleaning flow information to reduce memory footprint

* rename to more explicit name

* use immutable view instead of copy

* filter file types to n4js relevant types

* support counting of transpiled or deleted files during build

* use data of generated files instead of scan file system

* disable renaming of old hlc

* add debug outputs

* more debug messages

* more debug messages 2

* more debug messages 3

* more debug messages 4

* next try

* next try

* next try

* more debug messages 5

* more debug messages 6

* catch Throwables

* more debug messages 7

* more debug messages 8

* reverted debug output, SystemExitRedirector. Incorporate review feedback

* revert deletion

* add tests for ProjectDiscoveryHelper

* unset tag
@szarnekow
Copy link
Contributor

I just saw this one:

exception while obtaining type from type system: A different resource with the URI 'file:///Users/zarnekow/projects/n4js/lsp/onlinepresence/node_modules/org.eclipse.n4js.mangelhaft/src/n4js/org/eclipse/n4js/mangelhaft/types/TestFunctionType.n4js' was already registered.
java.lang.IllegalStateException: A different resource with the URI 'file:///Users/zarnekow/projects/n4js/lsp/onlinepresence/node_modules/org.eclipse.n4js.mangelhaft/src/n4js/org/eclipse/n4js/mangelhaft/types/TestFunctionType.n4js' was already registered.
	at org.eclipse.xtext.resource.XtextResourceSet.registerURI(XtextResourceSet.java:137)
	at org.eclipse.xtext.resource.SynchronizedXtextResourceSet.registerURI(SynchronizedXtextResourceSet.java:45)
	at org.eclipse.xtext.resource.XtextResourceSet$ResourcesList.inverseAdd(XtextResourceSet.java:152)
	at org.eclipse.xtext.resource.XtextResourceSet$ResourcesList.inverseAdd(XtextResourceSet.java:146)
	at org.eclipse.emf.common.notify.impl.NotifyingListImpl.addUnique(NotifyingListImpl.java:286)
	at org.eclipse.xtext.resource.SynchronizedXtextResourceSet$1.addUnique(SynchronizedXtextResourceSet.java:355)
	at org.eclipse.xtext.resource.SynchronizedXtextResourceSet$1.addUnique(SynchronizedXtextResourceSet.java:84)
	at org.eclipse.emf.common.util.AbstractEList.add(AbstractEList.java:304)
	at org.eclipse.xtext.resource.SynchronizedXtextResourceSet$1.add(SynchronizedXtextResourceSet.java:105)
	at org.eclipse.xtext.resource.SynchronizedXtextResourceSet$1.add(SynchronizedXtextResourceSet.java:84)
	at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.createResource(ResourceSetImpl.java:435)
	at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.createResource(ResourceSetImpl.java:423)
	at org.eclipse.xtext.resource.SynchronizedXtextResourceSet.createResource(SynchronizedXtextResourceSet.java:32)
	at org.eclipse.n4js.internal.AbstractN4JSCore.loadModuleFromIndex(AbstractN4JSCore.java:89)
	at org.eclipse.n4js.resource.N4JSResource.doResolveProxy(N4JSResource.java:1083)
	at org.eclipse.n4js.utils.emf.ProxyResolvingEObjectImpl.eResolveProxy(ProxyResolvingEObjectImpl.java:32)
	at org.eclipse.n4js.ts.typeRefs.impl.ParameterizedTypeRefImpl.getDeclaredType(ParameterizedTypeRefImpl.java:207)
	at org.eclipse.n4js.typesystem.utils.RuleEnvironmentExtensions.hasSubstitutionFor(RuleEnvironmentExtensions.java:1094)
	at org.eclipse.n4js.typesystem.utils.GenericsComputer.addSubstitution(GenericsComputer.java:159)
	at org.eclipse.n4js.typesystem.utils.GenericsComputer.primAddSubstitutions(GenericsComputer.java:140)
	at org.eclipse.n4js.typesystem.utils.GenericsComputer.addSubstitutions(GenericsComputer.java:113)
	at org.eclipse.n4js.typesystem.utils.TypeSystemHelper.addSubstitutions(TypeSystemHelper.java:145)
	at org.eclipse.n4js.typesystem.TypeJudgment$TypeJudgmentSwitchForASTNodes.caseParameterizedPropertyAccessExpression(TypeJudgment.java:885)
	at org.eclipse.n4js.typesystem.TypeJudgment$TypeJudgmentSwitchForASTNodes.caseParameterizedPropertyAccessExpression(TypeJudgment.java:1)
	at org.eclipse.n4js.n4JS.util.N4JSSwitch.doSwitch(N4JSSwitch.java:1109)
	at org.eclipse.emf.ecore.util.Switch.doSwitch(Switch.java:53)
	at org.eclipse.emf.ecore.util.Switch.doSwitch(Switch.java:69)
	at org.eclipse.n4js.typesystem.TypeJudgment.doApply(TypeJudgment.java:217)
	at org.eclipse.n4js.typesystem.TypeJudgment.apply(TypeJudgment.java:201)
	at org.eclipse.n4js.typesystem.N4JSTypeSystem.use_type_judgment_from_PostProcessors(N4JSTypeSystem.java:105)
	at org.eclipse.n4js.postprocessing.AbstractProcessor.invokeTypeJudgmentToInferType(AbstractProcessor.java:93)
	at org.eclipse.n4js.postprocessing.TypeProcessor.typeNode2(TypeProcessor.java:152)
	at org.eclipse.n4js.postprocessing.TypeProcessor.typeNode(TypeProcessor.java:97)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processNode_postChildren(ASTProcessor.java:389)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processSubtree(ASTProcessor.java:250)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processSubtree(ASTProcessor.java:246)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processSubtree(ASTProcessor.java:246)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processSubtree(ASTProcessor.java:246)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processSubtree(ASTProcessor.java:246)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processSubtree(ASTProcessor.java:246)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processAST(ASTProcessor.java:187)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processAST(ASTProcessor.java:146)
	at org.eclipse.n4js.postprocessing.N4JSPostProcessor.postProcessN4JSResource(N4JSPostProcessor.java:121)
	at org.eclipse.n4js.postprocessing.N4JSPostProcessor.performPostProcessing(N4JSPostProcessor.java:80)
	at org.eclipse.n4js.resource.PostProcessingAwareResource.performPostProcessing(PostProcessingAwareResource.java:226)
	at org.eclipse.n4js.resource.N4JSResource.performPostProcessing(N4JSResource.java:1369)
	at org.eclipse.n4js.resource.PostProcessingAwareResource.resolveLazyCrossReferences(PostProcessingAwareResource.java:153)
	at org.eclipse.n4js.resource.N4JSResource.resolveLazyCrossReferences(N4JSResource.java:1379)
	at org.eclipse.xtext.EcoreUtil2.resolveLazyCrossReferences(EcoreUtil2.java:504)
	at org.eclipse.n4js.ide.xtext.server.build.XStatefulIncrementalBuilder.buildClustured(XStatefulIncrementalBuilder.java:168)
	at org.eclipse.n4js.ide.xtext.server.build.XStatefulIncrementalBuilder.lambda$2(XStatefulIncrementalBuilder.java:125)
	at org.eclipse.xtext.xbase.lib.internal.FunctionDelegate.apply(FunctionDelegate.java:42)
	at com.google.common.collect.Lists$TransformingRandomAccessList$1.transform(Lists.java:612)
	at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
	at java.base/java.util.AbstractCollection.toArray(AbstractCollection.java:144)
	at java.base/java.util.ArrayList.addAll(ArrayList.java:701)
	at org.eclipse.n4js.ide.xtext.server.build.XClusteringStorageAwareResourceLoader.executeClustered(XClusteringStorageAwareResourceLoader.java:71)
	at org.eclipse.n4js.ide.xtext.server.build.XBuildContext.executeClustered(XBuildContext.java:59)
	at org.eclipse.n4js.ide.xtext.server.build.XStatefulIncrementalBuilder.launch(XStatefulIncrementalBuilder.java:124)
	at org.eclipse.n4js.ide.xtext.server.build.XIncrementalBuilder.build(XIncrementalBuilder.java:58)
	at org.eclipse.n4js.ide.xtext.server.build.XIncrementalBuilder.build(XIncrementalBuilder.java:39)
	at org.eclipse.n4js.ide.xtext.server.XProjectManager.doIncrementalBuild(XProjectManager.java:181)
	at org.eclipse.n4js.ide.xtext.server.XProjectManager.doInitialBuild(XProjectManager.java:130)
	at org.eclipse.n4js.ide.xtext.server.XBuildManager$1BuildInitialProjectJob.runJob(XBuildManager.java:154)
	at org.eclipse.n4js.ide.xtext.server.ParallelBuildManager$ParallelJob.run(ParallelBuildManager.java:40)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
exception while obtaining type from type system: The feature 'topLevelTypes' is not a valid feature
java.lang.IllegalArgumentException: The feature 'topLevelTypes' is not a valid feature
	at org.eclipse.emf.ecore.impl.BasicEObjectImpl.eStructuralFeature(BasicEObjectImpl.java:733)
	at org.eclipse.emf.ecore.impl.BasicEObjectImpl.eObjectForURIFragmentSegment(BasicEObjectImpl.java:527)
	at org.eclipse.emf.ecore.resource.impl.ResourceImpl.getEObject(ResourceImpl.java:811)
	at org.eclipse.emf.ecore.resource.impl.ResourceImpl.getEObject(ResourceImpl.java:787)
	at org.eclipse.xtext.resource.XtextResource.access$101(XtextResource.java:63)
	at org.eclipse.xtext.resource.XtextResource$1.getEObject(XtextResource.java:119)
	at org.eclipse.xtext.resource.DefaultFragmentProvider.getEObject(DefaultFragmentProvider.java:28)
	at org.eclipse.xtext.resource.XtextResource.basicGetEObject(XtextResource.java:363)
	at org.eclipse.xtext.resource.XtextResource.getEObject(XtextResource.java:349)
	at org.eclipse.xtext.linking.lazy.LazyLinkingResource.getEObject(LazyLinkingResource.java:246)
	at org.eclipse.xtext.resource.persistence.StorageAwareResource.getEObject(StorageAwareResource.java:119)
	at org.eclipse.n4js.resource.N4JSResource.getEObject(N4JSResource.java:1133)
	at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getEObject(ResourceSetImpl.java:223)
	at org.eclipse.n4js.resource.N4JSResource.doResolveProxy(N4JSResource.java:1093)
	at org.eclipse.n4js.utils.emf.ProxyResolvingEObjectImpl.eResolveProxy(ProxyResolvingEObjectImpl.java:32)
	at org.eclipse.n4js.ts.typeRefs.impl.ParameterizedTypeRefImpl.getDeclaredType(ParameterizedTypeRefImpl.java:207)
	at org.eclipse.n4js.typesystem.SubstTypeVariablesJudgment$SubstTypeVariablesSwitch.caseParameterizedTypeRef(SubstTypeVariablesJudgment.java:381)
	at org.eclipse.n4js.typesystem.SubstTypeVariablesJudgment$SubstTypeVariablesSwitch.caseParameterizedTypeRef(SubstTypeVariablesJudgment.java:1)
	at org.eclipse.n4js.ts.typeRefs.util.TypeRefsSwitch.doSwitch(TypeRefsSwitch.java:193)
	at org.eclipse.emf.ecore.util.Switch.doSwitch(Switch.java:53)
	at org.eclipse.emf.ecore.util.Switch.doSwitch(Switch.java:69)
	at org.eclipse.n4js.typesystem.SubstTypeVariablesJudgment$SubstTypeVariablesSwitch.substTypeVariables(SubstTypeVariablesJudgment.java:100)
	at org.eclipse.n4js.typesystem.SubstTypeVariablesJudgment$SubstTypeVariablesSwitch.substTypeVariables(SubstTypeVariablesJudgment.java:94)
	at org.eclipse.n4js.typesystem.SubstTypeVariablesJudgment$SubstTypeVariablesSwitch.internalSubstTypeVariablesInWildcard(SubstTypeVariablesJudgment.java:162)
	at org.eclipse.n4js.typesystem.SubstTypeVariablesJudgment$SubstTypeVariablesSwitch.caseWildcard(SubstTypeVariablesJudgment.java:115)
	at org.eclipse.n4js.typesystem.SubstTypeVariablesJudgment$SubstTypeVariablesSwitch.caseWildcard(SubstTypeVariablesJudgment.java:1)
	at org.eclipse.n4js.ts.typeRefs.util.TypeRefsSwitch.doSwitch(TypeRefsSwitch.java:258)
	at org.eclipse.emf.ecore.util.Switch.doSwitch(Switch.java:53)
	at org.eclipse.emf.ecore.util.Switch.doSwitch(Switch.java:69)
	at org.eclipse.n4js.typesystem.SubstTypeVariablesJudgment.apply(SubstTypeVariablesJudgment.java:62)
	at org.eclipse.n4js.typesystem.SubstTypeVariablesJudgment$SubstTypeVariablesSwitch.substTypeVariables(SubstTypeVariablesJudgment.java:102)
	at org.eclipse.n4js.typesystem.SubstTypeVariablesJudgment$SubstTypeVariablesSwitch.caseParameterizedTypeRef(SubstTypeVariablesJudgment.java:438)
	at org.eclipse.n4js.typesystem.SubstTypeVariablesJudgment$SubstTypeVariablesSwitch.caseParameterizedTypeRef(SubstTypeVariablesJudgment.java:1)
	at org.eclipse.n4js.ts.typeRefs.util.TypeRefsSwitch.doSwitch(TypeRefsSwitch.java:193)
	at org.eclipse.emf.ecore.util.Switch.doSwitch(Switch.java:53)
	at org.eclipse.emf.ecore.util.Switch.doSwitch(Switch.java:69)
	at org.eclipse.n4js.typesystem.SubstTypeVariablesJudgment.apply(SubstTypeVariablesJudgment.java:62)
	at org.eclipse.n4js.typesystem.N4JSTypeSystem.substTypeVariablesWithPartialCapture(N4JSTypeSystem.java:370)
	at org.eclipse.n4js.typesystem.N4JSTypeSystem.substTypeVariablesWithPartialCapture(N4JSTypeSystem.java:341)
	at org.eclipse.n4js.typesystem.TypeJudgment$TypeJudgmentSwitchForASTNodes.caseParameterizedPropertyAccessExpression(TypeJudgment.java:961)
	at org.eclipse.n4js.typesystem.TypeJudgment$TypeJudgmentSwitchForASTNodes.caseParameterizedPropertyAccessExpression(TypeJudgment.java:1)
	at org.eclipse.n4js.n4JS.util.N4JSSwitch.doSwitch(N4JSSwitch.java:1109)
	at org.eclipse.emf.ecore.util.Switch.doSwitch(Switch.java:53)
	at org.eclipse.emf.ecore.util.Switch.doSwitch(Switch.java:69)
	at org.eclipse.n4js.typesystem.TypeJudgment.doApply(TypeJudgment.java:217)
	at org.eclipse.n4js.typesystem.TypeJudgment.apply(TypeJudgment.java:201)
	at org.eclipse.n4js.typesystem.N4JSTypeSystem.use_type_judgment_from_PostProcessors(N4JSTypeSystem.java:105)
	at org.eclipse.n4js.postprocessing.AbstractProcessor.invokeTypeJudgmentToInferType(AbstractProcessor.java:93)
	at org.eclipse.n4js.postprocessing.TypeProcessor.typeNode2(TypeProcessor.java:152)
	at org.eclipse.n4js.postprocessing.TypeProcessor.typeNode(TypeProcessor.java:97)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processNode_postChildren(ASTProcessor.java:389)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processSubtree(ASTProcessor.java:250)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processSubtree(ASTProcessor.java:246)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processSubtree(ASTProcessor.java:246)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processSubtree(ASTProcessor.java:246)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processAST(ASTProcessor.java:187)
	at org.eclipse.n4js.postprocessing.ASTProcessor.processAST(ASTProcessor.java:146)
	at org.eclipse.n4js.postprocessing.N4JSPostProcessor.postProcessN4JSResource(N4JSPostProcessor.java:121)
	at org.eclipse.n4js.postprocessing.N4JSPostProcessor.performPostProcessing(N4JSPostProcessor.java:80)
	at org.eclipse.n4js.resource.PostProcessingAwareResource.performPostProcessing(PostProcessingAwareResource.java:226)
	at org.eclipse.n4js.resource.N4JSResource.performPostProcessing(N4JSResource.java:1369)
	at org.eclipse.n4js.resource.PostProcessingAwareResource.resolveLazyCrossReferences(PostProcessingAwareResource.java:153)
	at org.eclipse.n4js.resource.N4JSResource.resolveLazyCrossReferences(N4JSResource.java:1379)
	at org.eclipse.xtext.EcoreUtil2.resolveLazyCrossReferences(EcoreUtil2.java:504)
	at org.eclipse.n4js.ide.xtext.server.build.XStatefulIncrementalBuilder.buildClustured(XStatefulIncrementalBuilder.java:168)
	at org.eclipse.n4js.ide.xtext.server.build.XStatefulIncrementalBuilder.lambda$2(XStatefulIncrementalBuilder.java:125)
	at org.eclipse.xtext.xbase.lib.internal.FunctionDelegate.apply(FunctionDelegate.java:42)
	at com.google.common.collect.Lists$TransformingRandomAccessList$1.transform(Lists.java:612)
	at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
	at java.base/java.util.AbstractCollection.toArray(AbstractCollection.java:144)
	at java.base/java.util.ArrayList.addAll(ArrayList.java:701)
	at org.eclipse.n4js.ide.xtext.server.build.XClusteringStorageAwareResourceLoader.executeClustered(XClusteringStorageAwareResourceLoader.java:71)
	at org.eclipse.n4js.ide.xtext.server.build.XBuildContext.executeClustered(XBuildContext.java:59)
	at org.eclipse.n4js.ide.xtext.server.build.XStatefulIncrementalBuilder.launch(XStatefulIncrementalBuilder.java:124)
	at org.eclipse.n4js.ide.xtext.server.build.XIncrementalBuilder.build(XIncrementalBuilder.java:58)
	at org.eclipse.n4js.ide.xtext.server.build.XIncrementalBuilder.build(XIncrementalBuilder.java:39)
	at org.eclipse.n4js.ide.xtext.server.XProjectManager.doIncrementalBuild(XProjectManager.java:181)
	at org.eclipse.n4js.ide.xtext.server.XProjectManager.doInitialBuild(XProjectManager.java:130)
	at org.eclipse.n4js.ide.xtext.server.XBuildManager$1BuildInitialProjectJob.runJob(XBuildManager.java:154)
	at org.eclipse.n4js.ide.xtext.server.ParallelBuildManager$ParallelJob.run(ParallelBuildManager.java:40)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants