Found duplicate (but equal) classes in multiple 3.2.0 artifacts

[markkolich]

I upgraded my project from io.rest-assured:rest-assured version 3.1.1 to 3.2.0 and my builds started complaining of duplicate classes present in multiple artifacts:

[WARNING] Found duplicate (but equal) classes in [io.rest-assured:json-path:3.2.0, io.rest-assured:rest-assured-common:3.2.0, io.rest-assured:rest-assured:3.2.0, io.rest-assured:xml-path:3.2.0]:
[WARNING]   io.restassured.exception.PathException
[WARNING]   io.restassured.internal.assertion.AssertParameter
[WARNING]   io.restassured.internal.assertion.Assertion
[WARNING]   io.restassured.internal.assertion.AssertionSupport
[WARNING]   io.restassured.internal.assertion.EndToEndQuoteFragmentEscaper
[WARNING]   io.restassured.internal.assertion.GetAtPathFragmentEscaper
[WARNING]   io.restassured.internal.assertion.HyphenQuoteFragmentEscaper
[WARNING]   io.restassured.internal.assertion.PathFragmentEscaper
[WARNING]   io.restassured.internal.classpath.ClassPathResolver
[WARNING]   io.restassured.internal.mapper.ObjectDeserializationContextImpl
[WARNING]   io.restassured.internal.path.ObjectConverter
[WARNING]   io.restassured.mapper.DataToDeserialize
[WARNING]   io.restassured.mapper.ObjectDeserializationContext
[WARNING]   io.restassured.mapper.factory.ObjectMapperFactory
[WARNING]   io.restassured.mapper.resolver.ObjectMapperResolver
[WARNING] Found duplicate (but equal) classes in [io.rest-assured:json-path:3.2.0, io.rest-assured:rest-assured:3.2.0]:
[WARNING]   io.restassured.internal.path.json.ConfigurableJsonSlurper
[WARNING]   io.restassured.internal.path.json.JSONAssertion
[WARNING]   io.restassured.internal.path.json.JsonPrettifier
[WARNING]   io.restassured.internal.path.json.mapping.JsonObjectDeserializer
[WARNING]   io.restassured.internal.path.json.mapping.JsonPathGsonObjectDeserializer
[WARNING]   io.restassured.internal.path.json.mapping.JsonPathJackson1ObjectDeserializer
[WARNING]   io.restassured.internal.path.json.mapping.JsonPathJackson2ObjectDeserializer
[WARNING]   io.restassured.mapper.factory.DefaultGsonObjectMapperFactory
[WARNING]   io.restassured.mapper.factory.DefaultJackson1ObjectMapperFactory
[WARNING]   io.restassured.mapper.factory.DefaultJackson2ObjectMapperFactory
[WARNING]   io.restassured.mapper.factory.GsonObjectMapperFactory
[WARNING]   io.restassured.mapper.factory.Jackson1ObjectMapperFactory
[WARNING]   io.restassured.mapper.factory.Jackson2ObjectMapperFactory
[WARNING]   io.restassured.path.json.JsonPath
[WARNING]   io.restassured.path.json.config.JsonParserType
[WARNING]   io.restassured.path.json.config.JsonPathConfig
[WARNING]   io.restassured.path.json.exception.JsonPathException
[WARNING]   io.restassured.path.json.mapping.JsonPathObjectDeserializer
[WARNING] Found duplicate (but equal) classes in [io.rest-assured:rest-assured:3.2.0, io.rest-assured:xml-path:3.2.0]:
[WARNING]   io.restassured.internal.path.xml.GroovyNodeSerializer
[WARNING]   io.restassured.internal.path.xml.NodeBase
[WARNING]   io.restassured.internal.path.xml.NodeChildrenImpl
[WARNING]   io.restassured.internal.path.xml.NodeImpl
[WARNING]   io.restassured.internal.path.xml.XMLAssertion
[WARNING]   io.restassured.internal.path.xml.XmlEntity
[WARNING]   io.restassured.internal.path.xml.XmlPrettifier
[WARNING]   io.restassured.internal.path.xml.XmlRenderer
[WARNING]   io.restassured.internal.path.xml.mapping.XmlObjectDeserializer
[WARNING]   io.restassured.internal.path.xml.mapping.XmlPathJaxbObjectDeserializer
[WARNING]   io.restassured.mapper.factory.DefaultJAXBObjectMapperFactory
[WARNING]   io.restassured.mapper.factory.JAXBObjectMapperFactory
[WARNING]   io.restassured.path.xml.XmlPath
[WARNING]   io.restassured.path.xml.config.XmlParserType
[WARNING]   io.restassured.path.xml.config.XmlPathConfig
[WARNING]   io.restassured.path.xml.element.Node
[WARNING]   io.restassured.path.xml.element.NodeChildren
[WARNING]   io.restassured.path.xml.element.PathElement
[WARNING]   io.restassured.path.xml.exception.XmlPathException
[WARNING]   io.restassured.path.xml.mapping.XmlPathObjectDeserializer

Is this intentional?

CC // @larrysteinke

[johanhaleby]

Hmm it's definitely not intentional. When do you see this?

[markkolich]

@johanhaleby those warnings are from the duplicate-finder-maven-plugin, integrated into my project build:

https://github.com/basepom/duplicate-finder-maven-plugin

A quick peek at the raw .jar's does confirm that you have the same/duplicate classes in the mentioned artifacts.

This was not a problem in 3.1.1.

[markkolich]

@johanhaleby @ponziani it looks like this is perhaps caused by the usage of the maven-bundle-plugin, added for OSGi support with #1063.

If I'm reading this correctly [1], it looks like you're exporting classes from various packages which is effectively bundling/packaging those classes from each child into the other child JARs. Hence, the duplication.

For instance, this:

https://github.com/rest-assured/rest-assured/blame/cf3f6bd90069890220f2fa70ea0153625341cfd0/xml-path/pom.xml#L36-L41

...effectively copies (duplicates) classes from io.rest-assured:xml-path into io.rest-assured:rest-assured? A quick look at jar tvf rest-assured-3.3.0.jar on the command line confirms this finding: classes and resources from xml-path are inside of rest-assured.

That doesn't seem right.

Is this intentional?

[1] http://felix.apache.org/documentation/subprojects/apache-felix-maven-bundle-plugin-bnd.html#export-package

[ponziani]

It looks like the only good solution for this issue, is a refactoring to avoid split packages. After that, the statements can be refined to only contain the packages from the bundles itselves.

https://twitter.com/milendyankov/status/1095631397316976640

[markkolich]

A fun thread with @johanhaleby @ponziani and @azzazzel on Twitter indicates we may have found a possible solution:

https://twitter.com/milendyankov/status/1096093356227203077

[johanhaleby]

I'm ready to make an attempt at solving this but I'm not quite sure where to start.

@markkolich I had no idea that the bundle plugin copied the classes. I also don't know how to catch this in the build process. I've tried using duplicate-finder-maven-plugin but it doesn't give any hints related to this afaict.

"Sjarel" (could that be @ponziani?) suggested this on Twitter:

with that info we can suggest a rewrite to @johanhaleby. Avoiding split packages will break the API, but seems like the only correct solution imho.

But what exactly do I need to rewrite?

The json-path project is supposed to export io.restassured.path.json and the xml-path project is supposed to export io.restassured.path.xml. The internal package in both libraries should not be exposed (atleast not to the end users).

The rest-assured project doesn't have a package called io.restassured.path so I don't see what the problem is? Is it because all three projects contains the io.restassured.internal package (which shouldn't be exposed in any project)?

[markkolich]

I also don't know how to catch this in the build process. I've tried using duplicate-finder-maven-plugin but it doesn't give any hints related to this afaict.

I got it to fail quite handily with a mvn clean package by adding the plugin to your base pom.xml, and child modules.

Here's a draft PR demonstrating the approach:
#1135

The rest-assured project doesn't have a package called io.restassured.path so I don't see what the problem is?

Exactly – that is the problem. If io.rest-assured:rest-assured doesn't contain a io.restassured.path package, why are classes from that package in the resulting JAR rest-assured-3.3.0.jar? In theory, they shouldn't be, as classes in package io.restassured.path are provided by xml-path and json-path, completely separate artifacts.

As a result, the duplicate-finder-maven-plugin is complaining: hey you've got the same .class file in rest-assured-3.3.0.jar and xml-path-3.3.0.jar. In other words, rest-assured-3.3.0.jar contains copies of .class files found in other JARs on your classpath: in xml-path-3.3.0.jar.

The duplicate-finder-maven-plugin complains loudly of this, as this is usually a sign of a dependency hygiene problem. However, I don't know enough about OSGi to comment on whether or not this is safe to do in this context.

If it's really required to copy all classes in this project into a single JAR for OSGi support, perhaps repurposing rest-assured-all to be that JAR is an option? Sounds like @ponziani and @azzazzel have some thoughts on this.

[johanhaleby]

@markkolich Thanks for your comments. I see now that I actually did get warnings (but the build didn't fail which I expected it to do so this is why I didn't notice anything). Could you make your PR a non-draft PR so that I can merge it?

Exactly – that is the problem. If io.rest-assured:rest-assured doesn't contain a io.restassured.path package, why are classes from that package in the resulting JAR rest-assured-3.3.0.jar? In theory, they shouldn't be, as classes in package io.restassured.path are provided by xml-path and json-path, completely separate artifacts.

During the dist phase (when I make a release) I deliberately squash xml-path, json-path, rest-assured and rest-assured-common together into a "full jar" which is available from the Download page as rest-assured-3.3.0-dist.zip. This artefact includes both the sources, javadoc and class files and is generated for convenience purposes for people not using Maven et al as a build system. This done by the dist-rest-assured project. But this has been the case for as long as I can remember (at least 8 years or so) and I'm pretty confident that this hasn't affected the artifacts published to Maven central before. Other than that there's zero reasons for merging them together (unless it's required for some reason by OSGi or JPMS) and this has never been my intention.

If it's really required to copy all classes in this project into a single JAR for OSGi support, perhaps repurposing rest-assured-all to be that JAR is an option? Sounds like @ponziani and @azzazzel have some thoughts on this.

This sounds much better if it really is required, but I strongly suspect that it isn't. I'd be happy to remove the rest-assured-all project altogether if we can find another solution.

Anyway I'm truly grateful for all your help!

[markkolich]

Could you make your PR a non-draft PR so that I can merge it?

Sure thing; done!

But this has been the case for as long as I can remember (at least 8 years or so) and I'm pretty confident that this hasn't affected the artifacts published to Maven central before.

Understood. The dist task you're running to create rest-assured-3.3.0-dist.zip is a separate process from the step that publishes your artifacts to Maven Central. The problem here is specific to your artifacts on Maven Central. In short, the problem isn't with the "fat ZIP" you create for offline consumption, the problem is with the artifacts you're publishing to Maven Central, starting with version 3.2.0.

This was not a problem in 3.1.1. When I upgraded to rest-assured 3.2.0, that's when the duplicate-finder-maven-plugin in my project began complaining of duplicate classes in multiple rest-assured JARs as pulled from Maven Central.

The cause of that appears to be #1063.

[azzazzel]

Folks, I can try to clone and play with the project during the weekend. The solution seems to be rather simple (from pure OSGi perspective). However I'm not familiar with the project itself so I would need your help. Are there any tests that actually test the project behaves properly in OSGi environment? If so can you please provide some details how to run them. If not, can someone please explain what is expected when using the project in OSGi env.

[johanhaleby]

@azzazzel Would be great if you could have a look. There are tests in the /examples/rest-assured-itest-java-osgi project. They are activited with the osgi-tests profile. I.e. you need to run mvn clean install -Posgi-tests. The reason for why they are not activated by default (if I remember it correctly) was that the Felix framework didn't work with Java 11.

The osgi tests are executed by travis automatically (on oraclejdk8) on every commit.

[ponziani]

I'm only now able to answer this thread, sorry about showing up late. Some thoughts :

• Yes, my Twitter handle is Sjarel.
• The latest Felix release notes (6.0.2) don't mention anything about java 11 yet, I guess we'll have to wait a little longer for that.
• The fix looks good to me, but shouldn't you raise your major version number now ?

@azzazzel Too bad you're not familiar with rest-assured yet, it's really great for testing REST-services. Thanks for your help ! Big fan of your talks btw...

[johanhaleby]

@azzazzel @ponziani @markkolich This is probably not the right forum but I'm trying to upgrade to Groovy 2.5.6 and the OSGi tests fails:

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.586 sec <<< FAILURE! - in io.restassured.test.osgi.XmlPathOSGiITest
getUUIDParsesAStringResultToUUID(io.restassured.test.osgi.XmlPathOSGiITest)  Time elapsed: 2.567 sec  <<< ERROR!
java.io.IOException: Error resolving artifact org.codehaus.groovy:groovy-all:jar:2.5.6: Could not find artifact org.codehaus.groovy:groovy-all:jar:2.5.6 in central (http://repo1.maven.org/maven2/)

I don't understand why changing the version from 2.4.15 to 2.5.6 (that does indeed exist in maven central albeit the error message seem to indicate that it doesn't) would generate this error. Any ideas on how I can fix it?

[azzazzel]

@johanhaleby Could it be because you are expecting a JAR (org.codehaus.groovy:groovy-all:jar:2.5.6) and what is in central is actually POM (https://search.maven.org/artifact/org.codehaus.groovy/groovy-all/2.5.6/pom) ?

[johanhaleby]

@azzazzel Thanks for the tip. But from what I can tell there also seems to be a jar: https://search.maven.org/artifact/org.codehaus.groovy/groovy-all/2.5.6/jar

[johanhaleby]

@azzazzel But if I browse the repository I see that you're right. I guess they've removed this jar for some reason. Need to figure out how to upgrade properly. Thanks again for the hint!

[ponziani]

Looks like Groovy did quite a refactoring, I'm surprised they didn't up their major version number. That's exactly why the Maven property "groovy.range" has been added to RestAssured by the way.

Anyway, I tried to replace the groovy-all dependency by

mavenBundle().groupId("org.codehaus.groovy").artifactId("groovy").versionAsInProject(), mavenBundle().groupId("org.codehaus.groovy").artifactId("groovy-json").versionAsInProject().noStart(), mavenBundle().groupId("org.codehaus.groovy").artifactId("groovy-xml").versionAsInProject().noStart(),

(The 2nd and 3rd bundle are Fragments which cannot be started)

Unfortunately I don't know how to handle the error I'm getting now

org.osgi.framework.BundleException: Unable to resolve io.rest-assured.json-path [26](R 26.0): missing requirement [io.rest-assured.json-path [26](R 26.0)] osgi.wiring.package; (&(osgi.wiring.package=groovy.json )(version>=2.5.0)(!(version>=2.6.0))) [caused by: Unable to resolve groovy-json [19](R 19.0): missing requirement [groovy-json [19](R 19.0)] osgi.wiring.host; (&(osgi.wiring.host=groovy)(bundle-version>=0.0.0)) ] Unresolved requirements: [[io.rest-assured.json-path [26](R 26.0)] osgi.wiring.package; (&(osgi.wiring.package=groovy.json)(version>=2.5.0)(!(version>=2.6.0)))]

I'm assuming the Fragment host (The 1st bundle I added) cannot be found, but I don't understand why.

I succesfully tried accessing a package from the host bundle in the test itself and that works. By getting the groovy-json's state, I figured out it's INSTALLED, but I guess it should be RESOLVED.

[azzazzel]

I'm sorry I don't have the time to play with it now. But basicaly the error says that "io.rest-assured.json-path" needs a package "groovy.json" which is in version range 2.5.0 >= x > 2.6.0 and can not find such package in the OSGi runtime.

Most likely this is because what is on the compile classpath (which is where bnd gets the package versions from to put it in the metadata) is different from what you provide at runtime (in PaxExam statements above).

[ponziani]

I've tried the groovy bundle and groovy-json bundle into a standard Apache Felix and it turns out the groovy-json bundle is in status INSTALLED as well, with no logging to explain what went wrong. AFAIK a Fragment bundle should always get to status RESOLVED (is that correct @azzazzel ?), so I can only assume there is something wrong with the groovy-json bundle.

[azzazzel]

Ah, this one was actually a tricky one ;)
The thing is groovy-json uses Java's service loader. It has a requirement

Require-Capability: osgi.extender;filter:="(osgi.extender=osgi.serviceloader.registrar)",...

that prevents the bundle from resolving unless there is support for service loader. Sadly neither Felix nor Pax Exam report the error properly!

The OSGi support for service loader is provided by Aries SPI Fly project and thus one needs this bundle and the bundles providing the packages/services it depends on.

At the end of the day this is what you need to make the OSGi tests pass:

mavenBundle().groupId("org.apache.aries.spifly").artifactId("org.apache.aries.spifly.dynamic.bundle").version("1.2.1"),
mavenBundle().groupId("org.hamcrest").artifactId("hamcrest").version("2.1"),

mavenBundle().groupId("org.codehaus.groovy").artifactId("groovy").version("2.5.6"),
mavenBundle().groupId("org.codehaus.groovy").artifactId("groovy-json").version("2.5.6").noStart(),
mavenBundle().groupId("org.codehaus.groovy").artifactId("groovy-xml").version("2.5.6").noStart(),

[ponziani]

Impressive ! I read about this Groovy issue, but didn't see the connection with this Rest-Assured issue.

After handling a Hamcrest problem I managed to get all 3 integration tests working again and created a PR.

Do you feel we have enough info here to create an issue on the Felix bug tracker about the absence of proper error logging ?

[azzazzel]

I discussed this with some far more knowledgable OSGi folks and it seams Felix behaves according to the specs. That sad, it's worth opening a feature request with Felix project to provide more detailed information for while fragments are not resolved. Technically the information should be there (in the form of Exception) it's just a matter of figuring out a way to properly deliver it to the user.

[ponziani]

https://issues.apache.org/jira/browse/FELIX-6087