Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Gradle's Worker API #32

Merged
merged 9 commits into from Apr 7, 2019

Conversation

larsgrefer
Copy link
Contributor

@larsgrefer larsgrefer commented Mar 14, 2019

This removes the need for the classloading hack mentioned here:

// TODO very ugly class loading hack but as using org.gradle.process.internal.WorkerProcess does not work due to classpath problems using my own classes, this is the only why for now :-(

As addition, the task can now run parallel to other tasks, and multiple reports can be generated at once.

See also: https://guides.gradle.org/using-the-worker-api/

@aaschmid
Copy link
Owner

@larsgrefer very cool :-)

Can you agree to the terms of the Gradle CPD plugin Contributor License Agreement, also for this PR?

I will review it asap.

@larsgrefer
Copy link
Contributor Author

@aaschmid Some of the Tests in de.aaschmid.gradle.plugins.cpd.test.CpdAcceptanceTest aren't working, because it's not possible anymore to just call Task.execute(). These tests should be moved to de.aaschmid.gradle.plugins.cpd.test.CpdIntegrationTest and rewritten to use the Gradle TestKit (org.gradle.testkit.runner.GradleRunner).

@aaschmid
Copy link
Owner

@larsgrefer hm ... yes I am aware of that change - seems that Gradle does not like unit tests ... However, another way is to get the actions of the task and execute these actions. Do you want to try?

@aaschmid
Copy link
Owner

@larsgrefer Thanks for the work. It looks very well, already. Please don't be angry / afraid about all the review comments, I am happy to discuss them or even implement them myself if you prefer :-)

@larsgrefer
Copy link
Contributor Author

@larsgrefer hm ... yes I am aware of that change - seems that Gradle does not like unit tests ... However, another way is to get the actions of the task and execute these actions. Do you want to try?

Some of the tests (mostly the ones testing successful cases) could be refactored to directly use the new CpdAction, but some tests which actually test the task itself, should be refactored to use Gradle's TestKit.

src/main/groovy/de/aaschmid/gradle/plugins/cpd/Cpd.groovy Outdated Show resolved Hide resolved
languageProperties,
isSkipLexicalErrors(),
isSkipDuplicateFiles(),
new HashSet<>(getSource().getFiles()),
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to hand over an Collections.unmodifiableSet?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be possible, but what would be the benefit?

IMHO an unmodifiable set is only interesting for sets which are used for a long time and which are accessed/shared by multiple components.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm ... I thought more about the performance as copying the set is more expensive, even more on hugh projects ...

isSkipLexicalErrors(),
isSkipDuplicateFiles(),
new HashSet<>(getSource().getFiles()),
new ArrayList(enabledReports),
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some here ...

reporter.generate(matches)
logResult(matches)
}
if (isIgnoreLiterals()) {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can these be always set using Boolean.toString as for Tokenizer.OPTION_SKIP_BLOCKS?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure, I've just took the logic from here:

if (task.getIgnoreLiterals()) {
p.setProperty(Tokenizer.IGNORE_LITERALS, "true");
}
if (task.getIgnoreIdentifiers()) {
p.setProperty(Tokenizer.IGNORE_IDENTIFIERS, "true");
}
if (task.getIgnoreAnnotations()) {
p.setProperty(Tokenizer.IGNORE_ANNOTATIONS, "true");
}
p.setProperty(Tokenizer.OPTION_SKIP_BLOCKS, Boolean.toString(task.getSkipBlocks()));
p.setProperty(Tokenizer.OPTION_SKIP_BLOCKS_PATTERN, task.getSkipBlocksPattern());

src/main/groovy/de/aaschmid/gradle/plugins/cpd/Cpd.groovy Outdated Show resolved Hide resolved
} else {
throw new GradleException(message);
}
if (enabledReports.isEmpty()) {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm ... we assume that it is no use case to only have a console output without details in a report, don't we?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test "wants" the task to throw an exception in this case, thats why I built it that way.

def "executing 'Cpd' task throws wrapped 'InvalidUserDataException' if no report is enabled"() {
given:
project.cpdCheck{
reports{
csv.enabled = false
text.enabled = false
xml.enabled = false
}
source = testFile('.')
}
when:
project.tasks.getByName('cpdCheck').execute()
then:
!project.file('build/reports/cpdCheck.csv').exists()
def e = thrown(TaskExecutionException)
e.cause instanceof InvalidUserDataException
e.cause.message == '''Task 'cpdCheck' requires exactly one report to be enabled but was: [].'''
}

}
else {
String message = "CPD found duplicate code.";
SingleFileReport report = reports.get(0);
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should always return a report with the check and throw in Cpd.

src/main/groovy/de/aaschmid/gradle/plugins/cpd/Cpd.groovy Outdated Show resolved Hide resolved
String message = "CPD found duplicate code.";
SingleFileReport report = reports.get(0);
if (report != null) {
String reportUrl = report.getDestination().toString();
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about the clickabeFileUrl?

@@ -130,28 +130,6 @@ class CpdAcceptanceTest extends BaseSpec {
e.cause.message == '''Task 'cpdCheck' requires exactly one report to be enabled but was: [].'''
}

def "executing 'Cpd' task throws wrapped 'InvalidUserDataException' if more than one report is enabled"() {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to replace with an (integration-)test which enables two outputs and verify this?

@aaschmid aaschmid self-requested a review April 6, 2019 13:45
Copy link
Owner

@aaschmid aaschmid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @larsgrefer. Would you mind to discuss my comments or change the code?

@aaschmid aaschmid marked this pull request as ready for review April 7, 2019 08:57
@aaschmid aaschmid changed the base branch from master to use-worker April 7, 2019 09:51
@aaschmid aaschmid merged commit 115aa79 into aaschmid:use-worker Apr 7, 2019
@aaschmid
Copy link
Owner

aaschmid commented Apr 7, 2019

After merging the code, tests fail with an exception like the following. I will investigate further ...

de.aaschmid.gradle.plugins.cpd.test.CpdAcceptanceTest > applying 'Cpd' task to only parent project if only sub project has 'groovy' plugin FAILED
    org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':cpdCheck'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
        at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
        at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
        at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
        at org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
        at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
        at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:301)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:293)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:175)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
        at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
        at org.gradle.api.internal.AbstractTask.execute(AbstractTask.java:402)
        at de.aaschmid.gradle.plugins.cpd.test.CpdAcceptanceTest.applying 'Cpd' task to only parent project if only sub project has 'groovy' plugin(CpdAcceptanceTest.groovy:206)
        Caused by:
        org.gradle.workers.internal.DefaultWorkerExecutor$WorkExecutionException: A failure occurred while executing de.aaschmid.gradle.plugins.cpd.internal.CpdAction
            at org.gradle.workers.internal.DefaultWorkerExecutor.submit(DefaultWorkerExecutor.java:87)
            at de.aaschmid.gradle.plugins.cpd.Cpd.run(Cpd.groovy:203)
            at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:73)
            at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:46)
            at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:39)
            at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:26)
            at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:801)
            at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:768)
            at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$1.run(ExecuteActionsTaskExecuter.java:131)
            at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:301)
            at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:293)
            at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:175)
            at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
            at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
            at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:120)
            at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:99)
            ... 22 more
            Caused by:
            org.gradle.workers.internal.SerializingActionExecutionSpec$ParameterSerializationException: Could not serialize parameters
                at org.gradle.workers.internal.SerializingActionExecutionSpec.serialize(SerializingActionExecutionSpec.java:71)
                at org.gradle.workers.internal.SerializingActionExecutionSpec.<init>(SerializingActionExecutionSpec.java:42)
                at org.gradle.workers.internal.DefaultWorkerExecutor.submit(DefaultWorkerExecutor.java:85)
                ... 37 more
                Caused by:
                java.io.NotSerializableException: org.gradle.api.internal.provider.DefaultPropertyState
                    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
                    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
                    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
                    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
                    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
                    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
                    at java.util.ArrayList.writeObject(ArrayList.java:766)
                    at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1140)
                    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
                    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
                    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
                    at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
                    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
                    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
                    at org.gradle.workers.internal.SerializingActionExecutionSpec.serialize(SerializingActionExecutionSpec.java:69)
                    ... 39 more

@aaschmid
Copy link
Owner

aaschmid commented Apr 7, 2019

Problem is that properties of Cpd*FileReportImpl - which are used by Gradle - are not serializeable in the end. Therefore I build a mapping to simple DTOs for them. Still on it ...

@larsgrefer larsgrefer mentioned this pull request Apr 7, 2019
larsgrefer added a commit to larsgrefer/gradle-cpd-plugin that referenced this pull request Apr 7, 2019
@aaschmid
Copy link
Owner

aaschmid commented Apr 7, 2019

Problem is that properties of Cpd*FileReportImpl - which are used by Gradle - are not serializeable in the end. Therefore I build a mapping to simple DTOs for them.

Now a java.util.ServiceConfigurationError appears in most unit tests :-(
Even though integration test work, packaging the plugin and use it another projects fails it...

java.util.ServiceConfigurationError: net.sourceforge.pmd.cpd.Language: Provider net.sourceforge.pmd.cpd.CPPLanguage not a subtype
	at java.util.ServiceLoader.fail(ServiceLoader.java:239)
	at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:376)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	at net.sourceforge.pmd.cpd.LanguageFactory.<init>(LanguageFactory.java:27)
	at net.sourceforge.pmd.cpd.LanguageFactory.<clinit>(LanguageFactory.java:16)
	at de.aaschmid.gradle.plugins.cpd.internal.CpdAction.run(CpdAction.java:78)
	at org.gradle.workers.internal.DefaultWorkerServer.execute(DefaultWorkerServer.java:39)
	at org.gradle.workers.internal.DefaultWorkerServer.execute(DefaultWorkerServer.java:25)
	at org.gradle.workers.internal.IsolatedClassloaderWorkerFactory$WorkerCallable.call(IsolatedClassloaderWorkerFactory.java:177)
	at org.gradle.workers.internal.IsolatedClassloaderWorkerFactory.executeInWorkerClassLoader(IsolatedClassloaderWorkerFactory.java:105)
	at org.gradle.workers.internal.IsolatedClassloaderWorkerFactory.access$100(IsolatedClassloaderWorkerFactory.java:54)
	at org.gradle.workers.internal.IsolatedClassloaderWorkerFactory$1$1.call(IsolatedClassloaderWorkerFactory.java:78)
	at org.gradle.workers.internal.IsolatedClassloaderWorkerFactory$1$1.call(IsolatedClassloaderWorkerFactory.java:75)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:315)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:305)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:175)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:101)
	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
	at org.gradle.workers.internal.IsolatedClassloaderWorkerFactory$1.execute(IsolatedClassloaderWorkerFactory.java:75)
	at org.gradle.workers.internal.DefaultWorkerExecutor$1.call(DefaultWorkerExecutor.java:102)
	at org.gradle.workers.internal.DefaultWorkerExecutor$1.call(DefaultWorkerExecutor.java:96)
	at org.gradle.internal.work.AbstractConditionalExecution$1.run(AbstractConditionalExecution.java:38)
	at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner.runExecution(DefaultConditionalExecutionQueue.java:212)
	at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner.runBatch(DefaultConditionalExecutionQueue.java:161)
	at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner.run(DefaultConditionalExecutionQueue.java:130)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
	at java.lang.Thread.run(Thread.java:748)

aaschmid pushed a commit that referenced this pull request Apr 22, 2019
@aaschmid aaschmid added this to the v1.4 milestone Apr 22, 2019
aaschmid added a commit that referenced this pull request May 10, 2019
* use-worker-api:
  add JDKs 10 and 11 for travis-ci
  test minimal possible toolVersion and fallback language
  remove deprecated API
  adjust minimal required PMD version
  create executor configuration and move to dedicated package (#38)
  add check methods for state of task before actually execute anything
  use serializable report configuration for CpdAction (#38)
  remove no longer required excludes
  remove unnecessary loggers for report implementations
  Use Gradle's Worker API (#32)
  use 'getByName' consistent to other test cases
  remove @optional for non-optional properties
  add integration test which really executes CPD
  add both test sourceSets to gradlePlugin
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants