Gradle 7 requires duplicatesStrategy for "fake" duplicates - improve error message

[lkoe]

Gradle 7 rightfully requires a duplicatesStrategy for handling duplicate files.

However, the respective error can also be produced by adding the same directory to a source set multiple times.
This is a situation often encountered when using multiple third-party plugins, which add the same directories to a source set.

This situation is mostly out of control of the user, yet requires to specify a handling strategy - which in this case seems ultimately unnecessary, since no real file duplicates (duplicate files in multiple directories) are involved.

SourceDirectorySet#getSourceDirectories() might be a potential point, where duplicate source directories could be considered and filtered.

Expected Behavior

Gradle should allow innocently adding the same source directory multiple times, without requiring to set duplicatesStrategy for common tasks (processResources, sourcesJar)

Current Behavior

Gradle throws the duplicatesStrategy error, when the same source directory is added multiple times.

Context

We are using complex builds with a variety of third-party plugins with little to no control over them. The problem forced us to update our build scripts/custom plugins, even though there weren't any actual duplicates involved.

Steps to Reproduce

Build the attached sample project.
g7-duplicate-sources.zip

Your Environment

Gradle 7.0/7.0.2, Windows 10
Build scan URL:

[lackovic]

I am also facing this issue while upgrading from Gradle 6.6.1 to 7.0.2, with no real file duplicates involved:

Task :compileE2eTestJava
[Step 1/1] :compileE2eTestJava
[:compileE2eTestJava] FAILURE: Build failed with an exception.
[:compileE2eTestJava] 
[:compileE2eTestJava] * What went wrong:
[:compileE2eTestJava] Execution failed for task ':processE2eTestResources'.
[Step 1/1] > Entry application.properties is a duplicate but no duplicate handling strategy has been set. Please refer to https://docs.gradle.org/7.0.2/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details.
[Step 1/1] > Task :processE2eTestResources FAILED
[Step 1/1] * Try:
[Step 1/1] Run with --info or --debug option to get more log output. Run with --scan to get full insights.
[Step 1/1] :processE2eTestResources
[:processE2eTestResources] * Exception is:
[:processE2eTestResources] org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':processE2eTestResources'.

The note to refer to https://docs.gradle.org/7.0.2/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details is not helping since the task processE2eTestResources is defined by Gradle, not by me, so it is unclear to what exactly I need to add this property in order to fix this issue.

[donat]

This is a manifestation of a missing internal Gradle feature. Gradle does not do any deduplication in this use case. And even if such a feature would exist, this configuration would cause many issues. Tasks cannot be up-to-date, caching won't work, etc.

It is the plugin's responsibility to define unique input and output folders. If there's such an overlap, the plugins should also enable the clients to configure those folders.

[lkoe]

Agreed that this kind of overlap can be troublesome for other Gradle features, like you mentioned.

But the issue at hand is that Gradle is flagging duplicates, when in fact there aren't any. I still think this should be adressed.

[dimaKudr]

@donat from the Gradle user perspective it looks like a defect introduced in Gradle 7. It would be very nice for the Gradle team to mention it in the Upgrade Guide here https://docs.gradle.org/current/userguide/upgrading_version_6.html. Its current section about Duplication Strategy is kind of misleading. It says "encounters a duplicate entry", and when I read that, I think "but I don't have any duplicates, so it is not relevant to me". Hope you see my point.
And fix for the problem was suggested here https://discuss.gradle.org/t/gradle-7-fail-for-duplicates-in-copy-specs-has-no-duplicates-in-project/39834/8

[unoexperto]

Folks, what is the solution for this problem? I'm honestly baffled. I'm using 7.1. I followed advice from official upgrade documentation and added this

tasks {
    withType<Copy> {
        duplicatesStrategy = DuplicatesStrategy.EXCLUDE
    }
}

But it doesn't seem to help, as if this setting is not inherited by other tasks. So that distTar or assemble fail for me with the same error unless I set duplicatesStrategy for each of these tasks.

UPD: I did following hack, but guys, this is horrible.

gradle.taskGraph.whenReady {
    allTasks
        .filter { it.hasProperty("duplicatesStrategy") } // Because it's some weird decorated wrapper that I can't cast.
        .forEach {
            it.setProperty("duplicatesStrategy", "EXCLUDE")
        }
}

[mbwhite]

In case it helps. I believe I hit exactly this problem, and the solution that seems to work for me is

task sourcesJar(type: Jar) {
    duplicatesStrategy = 'include'
    classifier = 'sources'
    from sourceSets.main.allSource
}

java {
    withJavadocJar()
    withSourcesJar()
}

Order of these is important! task sourcesJar must come first, and you must also have withSourcesJar()

[ylemoigne]

For anyone coming here and using kotlin multiplatform, the plugin trigger the issue if withJava() is used : https://youtrack.jetbrains.com/issue/KT-46978

The official gradle response to this issue is very disapointing. It's ok to don't fix it as plugin should not do what they are doing.
But showing some level of empathy for your users would be greatly appreciated.

Actually we have a message wich talk about a duplicated resource that is not duplicated. It's missleading.
Their is nothing in documentation/upgrade-guide that can help.
Users can spend hours searching in documentation/web before finding this issue (or the forum post mentionned)
If this issue didn't exist or if user don't find it, they don't understand what happen, they feel stupid for their not being able to make their project work. Or they curse gradle for being "too complicated" and "not working".

This not very pleasant for everyone.

Once this issue is found, it's time to remove plugins, comment any build part depending on it, and re-add/enable them one by one, maybe in different combination to search culprit(s). On complex build, it's cumbersome.

What can be done ?
At level 1, mention in upgrade guide that some plugin which was incorrectly using copy might break the build.
At level 2, mention in the error message that it might be caused by plugin bugs.
At level 3, detect directory added multiple times, produce a message adapted to this situation
At level 4, detect directory added multiple times, and say which plugin(s)/call-site is adding it

I understand that it may be not trivial to handle.
But saying "not our problem : good bye" when issue is not directly under user control nor visibility of user and triggered by and update of gradle, letting user suffering from it, it feel like gradle don't care and it's just sad...

[robvadai]

@ylemoigne I completely agree.

Gradle guys, the issue is if Gradle gets too overwhelming to use it'll loose popularity. I don't want that, I love to use Gradle for many years now but this is a warning sign. Please make sure releases are user friendly.

I had genuine duplicate files in my repo (log4j.properties in different test directories) and my solution was:

allprojects {
  apply plugin: 'java'

  tasks.withType(Copy).all {
    duplicatesStrategy 'exclude'
  }
...

[lackovic]

I had limited time to troubleshoot my issue mentioned above so eventually I had to rollback to version 6.

A possible workaround (not tested by me yet), suggested by this comment, is to add the following to the build.gradle:

rootProject.tasks.named("processE2eTestResources") {
    duplicatesStrategy = DuplicatesStrategy.INCLUDE
}

[calvertdw]

Kotlin type safe syntax that worked for me:

project.tasks.named("processResources", Copy::class.java) {
   duplicatesStrategy = DuplicatesStrategy.INCLUDE
}

[GregMarshBYKA]

I am just getting my system set up with Kotlin and have encountered this with their "Create your first Kotlin application" tutorial. I was surprised that it did not work given I used all the defaults - thus "hello world" fails to create Jar :). Sad I know.

I need to provide a working tutorial to a group of coders who are just starting out. Did any of you find a simple solution to this issue? I would appreciate a few extra lines of explanation so that anyone new can follow it.
Thank you in advance.

The tutorial I am using is found at https://www.jetbrains.com/help/idea/create-your-first-kotlin-app.html#package-as-jar

*** Update / I went back to Gradle 6.9.1 and it worked like a charm. Hopefully the bug in 7+ is fixed. I tried 7.1 and 7.2 but both had the same issue.

[JiangHongTiao]

Facing the same issue after upgrade from 6.8 to 7.2. Also reverting to 6.9.1, as it works for me too.

[GregMarshBYKA]

@JiangHongTiao - I contacted jetbrains support and they pointed back to gradle so I see no point trying to come up with a work around with gradle for it does not make sense to put in extra code throughout a project just to fix a build issue.

Hope gradle/jetbrains figures out what is actually going wrong with 7 to 7.2 ....

[jhyry-gcpud]

Still having this issue in 7.2.

[GregMarshBYKA]

I really appreciate everyone's comments for it shows that this is clearly an issue and not something that is born out of a failure to understand how Gradle works.
That said, can anyone suggest an effective path to engage Gradle regarding this fundamental failure? Not knowing the inner workings of Gradle I do not want to assume that the solution is only dependent on resources being assigned to fix a bug. If it is complex and can't be fixed right now may we know why?
Like many support threads, it is very frustrating to watch users fix or attempt to fix things that should be dealt with by those in the know.
Fingers crossed.

[uap-universe]

@big-guy can you please confirm, that you are still working on this? @jvandort can you please have a look and maybe reassign this issue if @big-guy is currently busy doing something else? Thank you so much :-)

[fsparv]

If I understand the (quickly scanned) discussion above this indicates that some form of copy or similar task is writing a file that already exists. If that's true this has got to be the worst error message ever... "you have a duplicate Named X. Too bad, you figure it out" is about what it says. The great thing is that it's often the same thing being copied twice so no matter how much you grep or search for a duplicate, it's not there. Duplicate of what? where is the destination that seems to contain a duplicate? If gradle wants to ensure that one never accidentally overwrites via copy, then it needs to track all the things that copy files. (src/main/resources/foo.txt --> build/resourceTmp/foo.txt etc) Then when it tries to copy src/test/resources/foo.txt --> build/resourceTmp/foo.txt it can complain like this:

src/test/resources/foo.txt attempting to overwrite /build/resourceTmp/foo.txt (previously copied from src/main/resources/foo.txt) 

For even better clarity track the current task and modifed times so that one could understand

src/main/resources/foo.txt (task:mungeResources) attempting to overwrite /build/resourceTmp/foo.txt (previously copied from src/main/resources/foo.txt (task:copyResources) 

And possibly NOT fail if the mod time hadn't changed.

Wait, that's too much memory you say... possibly which is why the biggest problem here is this should be an optional feature for folks who are having issues with things getting mysteriously overwritten. Maybe something like

allProjects {
  strictCopy true
}

[notaLonelyDay]

Problem:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':mycoolmodule:processResources'.

Caused by: org.gradle.api.InvalidUserCodeException: Entry mycoolresource is a duplicate but no duplicate handling strategy has been set. Please refer to https://docs.gradle.org/7.5/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details.

Solution:
processResources.duplicatesStrategy = DuplicatesStrategy.EXCLUDE

[GregMarshBYKA]

Problem: org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':mycoolmodule:processResources'.

Caused by: org.gradle.api.InvalidUserCodeException: Entry mycoolresource is a duplicate but no duplicate handling strategy has been set. Please refer to https://docs.gradle.org/7.5/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details.

Solution: processResources.duplicatesStrategy = DuplicatesStrategy.EXCLUDE

Thank you but we are discussing a bug in Gradle whereby an error is thrown when a duplicate does not exist.

[GregMarshBYKA]

If I understand the (quickly scanned) discussion above this indicates that some form of copy or similar task is writing a file that already exists. If that's true this has got to be the worst error message ever... "you have a duplicate Named X. Too bad, you figure it out" is about what it says. The great thing is that it's often the same thing being copied twice so no matter how much you grep or search for a duplicate, it's not there. Duplicate of what? where is the destination that seems to contain a duplicate? If gradle wants to ensure that one never accidentally overwrites via copy, then it needs to track all the things that copy files. (src/main/resources/foo.txt --> build/resourceTmp/foo.txt etc) Then when it tries to copy src/test/resources/foo.txt --> build/resourceTmp/foo.txt it can complain like this:

src/test/resources/foo.txt attempting to overwrite /build/resourceTmp/foo.txt (previously copied from src/main/resources/foo.txt) 

For even better clarity track the current task and modifed times so that one could understand

src/main/resources/foo.txt (task:mungeResources) attempting to overwrite /build/resourceTmp/foo.txt (previously copied from src/main/resources/foo.txt (task:copyResources) 

And possibly NOT fail if the mod time hadn't changed.

Wait, that's too much memory you say... possibly which is why the biggest problem here is this should be an optional feature for folks who are having issues with things getting mysteriously overwritten. Maybe something like

allProjects {
  strictCopy true
}

Thank you but we are discussing a bug in Gradle whereby an error is thrown when a duplicate does not exist.

[D-Kumar19]

...is a duplicate but no duplicate handling strategy has been set. Please refer to https://docs.gradle.org/7.5/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details.

This is the problem that I am receiving when I try to build my project in which I am trying to upgrade the Gradle version from 6.7 to 7.5. I tried to add this in build.gradle but the error still persists.

allprojects {
    apply plugin: 'java'

    tasks.withType(Copy).all {
        duplicatesStrategy 'exclude'
    }
    ...

I also tried to add this in my build.gradle.in file but I am not able to remove this error and every time build fails. Even tried it with .INCLUDE and .EXCLUDE but no way to get rid of this error.
tasks.withType(Copy) { duplicatesStrategy = DuplicatesStrategy.WARN }

There are two build.gradle.in files but the error is from first file. I have tried to add into the first file in which there is error and also tried it in both files but no chance. Moreover, when I try to build this project locally it works fine but when I try to build it using Jenkins it fails. It works locally as it is using another image and using that image it works fine as Gradle version in that image is not updated.

The issue has been resolved I actually had to specify the files because of which the duplicate strategy error was being thrown. For example: Jar files.

tasks.withType(Jar){
    duplicatesStrategy = DuplicatesStrategy.INCLUDE
}

Thought it might help someone. Thanks

[spyro2000]

It's another magic line I will handle to my simple Spring Boot project to make it work with Kotlin and QueryDSL. Don't really know, what causes the actual problem but I am just tired of this.

[carstenSpraener]

is it possible this bug is still alive? Having some issues with gradle 8.1.1
see #25208

Any help is very welcome cause i can't publish at the moment. This is a show stopper

[GregMarshBYKA]

I don't know why you can't publish here carstenSpraener (I can't reference you either) - this itself is an issue but to your point, I have not seen anything published that refers to this being resolved and everything so far that I have seen is a workaround for a "NON ERROR".
I am hoping that something will be posted here or in the releases that will specifically address this issue for we are still using the old version of Gradle.

[poo0054]

案是：log4

I found that you are too right, I have been using the maven, but spring in the use of gradle. I would like to learn a little, I found that it is too dissuasive, no matter what, I never clone the code down after the direct start success, each time there will be a lot of small problems, and finally I really do not want to touch the

[Azahe]

I am not sure if anyone of the gradle team watches this task anymore - however maybe this can useful for others, our solution (in root build.gradle file):

subprojects {
  // To prevent naming conflicts in multi-module projects, include the full relative path in the jar name.
  // This is crucial when common module names (like 'api' or 'core') exist in various subdirectories.
  // This strategy ensures jar name uniqueness across the project.
  tasks.withType(Jar).tap {
    configureEach {
      archiveBaseName.set(project.path.replace(':', '-').stripMargin('-'))
    }
  }
}

In a situation where two sub modules have same name (as described in the comment) default value for jar files causes error in tasks where these are collected together (for example Spring's bootWar task which I assume many people relay on in one way or another).

This is direct result of Project#getName method returning potentially non unique value for multiproject file being used as a default for archive base name.
One potential solution would be to change that default to Project#getPath - which according to documentation returns unique value in the repository.

In this case meddling with duplicatesStrategy seems misguided - as allowing duplicates can lead to unintended side effects - like duplicates that developers don't know about.

This doesn't address conflicts from "outside-of-repo" jars but still looks like an overall improvement. Also changing : to - can maybe be skipped.

[GregMarshBYKA]

I am not sure if anyone of the gradle team watches this task anymore - however maybe this can useful for others, our solution (in root build.gradle file):

subprojects {
  // To prevent naming conflicts in multi-module projects, include the full relative path in the jar name.
  // This is crucial when common module names (like 'api' or 'core') exist in various subdirectories.
  // This strategy ensures jar name uniqueness across the project.
  tasks.withType(Jar).tap {
    configureEach {
      archiveBaseName.set(project.path.replace(':', '-').stripMargin('-'))
    }
  }
}

In a situation where two sub modules have same name (as described in the comment) default value for jar files causes error in tasks where these are collected together (for example Spring's bootWar task which I assume many people relay on in one way or another).

This is direct result of Project#getName method returning potentially non unique value for multiproject file being used as a default for archive base name. One potential solution would be to change that default to Project#getPath - which according to documentation returns unique value in the repository.

In this case meddling with duplicatesStrategy seems misguided - as allowing duplicates can lead to unintended side effects - like duplicates that developers don't know about.

This doesn't address conflicts from "outside-of-repo" jars but still looks like an overall improvement. Also changing : to - can maybe be skipped.

@Azahe , thank you for your code sample which addresses issues that I am sure people will benefit from. Unfortunately, the core issue here is not related - Gradle 7 broke something and flags duplicates when there aren't any. This is the real issue and as per your comment at the beginning, I do not know if anyone is looking at this for there does not seem to be any further input. I hope this topic stays open and is not closed due to lack of responses for as far as I can tell nothing has changed.

[lkoe]

I'm the orignal bug author and reported a very specific situation, where adding the same source directory twice to the project (something that multiple third-party plugins might do) leads to erroneous reporting of duplicates, as demonstrated by a reproducer project attached.
This specific misbehaviour/bug has been closed and flagged as no-fix by the gradle devs (#17236 (comment)). So there probably won't be happening anything regarding the original issue.

For some reason this specific report seems to be found by people dealing with all sorts of duplication issues and has become sort of a fly trap since.
I can only recommend to file a separate issue for your specific problem, ideally with a reproducer sample attached.

[GregMarshBYKA]

@lkoe Hi Lars, there are many variations added but the core issue that you pointed out resulting from the requirement to add a "duplicatesStrategy" even for nonexistent duplicates is still there and a lot of the comments relate so hopefully someone is taking note. Cheers

[ov7a]

Thanks, everyone, for the discussion and sorry for the late reply.

As @donat pointed out in his message, the idiomatic way to resolve this is to change Gradle plugins and scripts to avoid intersecting directories. The default fail-fast strategy is correct and, as some of you indicated, helpful in most cases, so it's unlikely that this behavior will be changed.

However, the error message should definitely be improved to indicate the sources of the duplicates to ease the debugging and fixing the issue.

Another option might be improving the duplicates strategy to ignore exact duplicates (e.g. with matching fingerprints) - this should solve the issue for newbie users. However, it's up to the team responsible for that area to decide.

---

The issue is in the backlog of the relevant team, but this area of Gradle is currently not a focus one, so it might take a while before a fix is made.