RFE: Duplicate checker

[vorburger]

I have used this library to implement a classpath duplicate checker. The gist of it is basically:

Map<String, List<String>> check() {
        Map<String, List<String>> seen = new HashMap<>();
        try (ScanResult scanResult = new ClassGraph().enableClassInfo().ignoreClassVisibility().scan()) {
            for (Resource resource : scanResult.getAllResources()) {
                String path = resource.getPath();
                @Var List<String> classpathElements = seen.get(path);
                if (classpathElements == null) {
                    classpathElements = new ArrayList<>(1);
                    seen.put(path, classpathElements);
                }
                classpathElements.add(getClasspathElementString(resource));
            }
        }
        return seen.entrySet().stream().filter(entry -> entry.getValue().size() > 1)
                .filter(entry -> !isHarmlessDuplicate(entry.getKey()))
                .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
    }

Basically I re-built something very similar to https://github.com/jhades/ (see http://jhades.github.io) or https://github.com/basepom/duplicate-finder-maven-plugin but using classgraph. (The motivation for this was that JHades could not support Java 9+.)

Filing this issue here to ask whether a contribution (PR) of utility class like my ClasspathHellDuplicatesChecker into this lib would be of any interest. Alternative I could just add a link to it somewhere in the doc to make it easier and let people build something like it.

Also absolutely no offense taken if this doesn't fit here - just close in that case, no worries.

[vorburger]

also mentioned on https://stackoverflow.com/a/52637869/421602

[lukehutch]

Hi @vorburger, thanks for this suggestion and code submission! I can see how this would be useful, and it should probably be part of the ResourceList API. Take a look at the commit above (basically the same as your code, but reworked a bit), and let me know if this works for you. To find duplicate classfiles, you would use:

new ClassGraph().scan().getAllResources().classFilesOnly().findDuplicatePaths()

Note that ClassGraph already does a lot of work to ignore duplicates:

• Duplicate jars or directories on the classpath are ignored (i.e. if a jar is listed twice, it is only scanned once)
• Duplicate classes (classes with a given name defined twice on the classpath, whether or not the classfile content is identical) are ignored (i.e. only the first instance of a given class definition on the classpath or module path is ever scanned, to mimic Java's classfile masking / shadowing logic)
• Duplicates of both classfiles and resource files within different version layers of a multi-release jar are filtered out, so that only the most highest version (less than or equal to the running JRE) is visible in a ClassInfoList or ResourceList -- i.e. META-INF/versions/11/abc.txt will mask META-INF/versions/9/abc.txt, which will mask abc.txt. (This also emulates the JRE's shadowing logic.)
• Duplicate resources are allowed, as long as the classpath element jar or directory itself is not duplicated (as described in the first point).
• However, classfiles are returned in ResourceList results, so detecting .class file extensions is the only way to get at duplicated classfiles (as you have noticed, I think). This is why I also added the method .classFilesOnly() and .nonClassFilesOnly(), to allow for filtering for these / for filtering them out.

Please let me know if this works for you!

[vorburger]

@lukehutch nice! I'll switch to using that, once you cut a release including this.

What this is missing is just the filtering of well-known harmless duplicates.

Do you want PR with my ClasspathHellDuplicatesCheckRule (and that filtering) ?

[vorburger]

also, perhaps after we have the JUnit Rule along, it would be neat to highlight this on the README?

[lukehutch]

@vorburger Unfortunately the list of "well-known" duplicates will never be complete, and it will be a neverending battle to keep it up to date as part of a library whose goal is to be as generic and general as ClassGraph strives to be. It's also conceivable that such a list could end up accidentally masking something that should be detected as a duplicate. I think this is better left up to users of ClassGraph to implement their own filters for their own specific classpaths / module path situations.

However, feel free to create a mini-library that depends on ClassGraph and wraps your duplicate filters, and even potentially implements a minimal subset of the JHades API so that this can serve as a drop-in replacement for that. It seems like that may be useful, since JHades development has stalled.

I have relased this as 4.2.7. Thanks again for the suggestion and for the code submission!

[lukehutch]

Release notes: https://github.com/classgraph/classgraph/releases/tag/classgraph-4.2.7

[lukehutch]

also, perhaps after we have the JUnit Rule along, it would be neat to highlight this on the README?

Yes, I updated the API docs to reflect the new API calls, and I'm about to add an example in the "Code examples" page too, since this is a useful capability.

[vorburger]

see the ClasspathHellDuplicatesChecker in OpenDaylight's infrautils and a brief related write up in http://blog2.vorburger.ch/2019/02/how-to-find-duplicate-classes-and.html

[lukehutch]

@vorburger this is great, but be aware of one thing: if the exact same path or URL (after canonicalization) is listed on the classpath or module path multiple times, the second and subsequent are ignored by ClassGraph (since that is more of less semantically what the JRE does too). Therefore, duplicates are only detected if the same resources are present in multiple different modules, jars or directories.

Maintaining isHarmlessDuplicate() over the years is going to be a Herculean effort, good luck :D

PS you should assign the ScanResult in a try-with-resources statement, so that proper cleanup does not depend upon a finalizer.

[vorburger]

if the exact same path or URL (after canonicalization) is listed on the classpath or module path multiple times, the second and subsequent are ignored by ClassGraph (since that is more of less semantically what the JRE does too). Therefore, duplicates are only detected if the same resources are present in multiple different modules, jars or directories.

Erm ... but that's perfectly fine, right? That's kind of the point of what I'm after - to find duplicate classes in different JARs, when you (or an external 3rd-party) made a mess of your dependencies. You can't really have a duplicate of the exact same class in the same single JAR.

Maintaining isHarmlessDuplicate() over the years is going to be a Herculean effort, good luck :D

I'm not planning to... 😈

PS you should assign the ScanResult in a try-with-resources statement, so that proper cleanup does not depend upon a finalizer.

Thanks for noticing and pointing it out - fixed in https://git.opendaylight.org/gerrit/#/c/80158.

[lukehutch]

Erm ... but that's perfectly fine, right? That's kind of the point of what I'm after - to find duplicate classes in different JARs, when you (or an external 3rd-party) made a mess of your dependencies. You can't really have a duplicate of the exact same class in the same single JAR.

Yes, it's fine, I just wanted to make the distinction between multiple listings of the same jar on the classpath, and multiple occurrences of the same file within different classpath elements. The first cannot be detected (but may still be considered by somebody to be problematic), but the second can.