Interoperability with Ivy and Maven dependency resolvers

[ckulenkampff]

This enhancement would allow to create a flat classpath of Ceylon CARs for Java projects using Maven, Ivy or Gradle. This is possible by offering appropriate repository "facades" through the Herd repository server.

Maven repository structure
Maven expects the following layout (see Maven Repository Layout - Final) for primary artifacts:
/$groupId[0]/../${groupId[n]/$artifactId/$version/$artifactId-$version.$extension
and for secondary artifacts:
/$groupId[0]/../$groupId[n]/$artifactId/$version/$artifactId-$version-$classifier.$extension

Ivy repository structure
Ivy is more flexible and allows to specify custom patterns for artifact resolution (see Ivy Documentation - Main Concepts). The default patten that is used by Gradle is the following (see Gradle DSL Reference - IvyArtifactRepository):
Artifacts: $baseUri/[organisation]/[module]/[revision]/[type]s/[artifact](.[ext])
Ivy module descriptors: $baseUri/[organisation]/[module]/[revision]/[type]s/[artifact](.[ext])

Meta information
To resolve transitive dependencies both repository types require meta information. Maven uses pom.xmls. Ivy uses ivy.xmls, but can also process pom.xmls. Those files must be accessible via HTTP requests.

Meta information augmentation
When the repository server responds to a "foreign" meta data request for a Ceylon module, it should automatically add all implicit dependencies of the Ceylon language to the response. For interoperability these Ceylon language modules should be published to the Herd repository so that Java projects that depend on a Ceylon library do not have to provide them by themselves.

Artifact aliases
For interoperability it would be very useful when CAR files are also available under the same name but with JAR file extension when accessed through a facade.

Many IDEs automatically link source and javadoc JARs to the downloaded artifacts by searching in the module cache for files like $artifactId-$version-$classifier-sources.$extension (IDE dependent see NetBeans DependencyNode). Ceylon source artifacts should be made available in a way that this resolution works out of the box. This means that the artifacts are made available under another name than they are normally accessible in Herd.

[FroMage]

I had the same thought the other day, I think that'd be really handy indeed. Would you be willing to test this for me?

[FroMage]

So Maven repo docs fail to mention these:

• http://central.maven.org/maven2/last_updated.txt (simple enough timestamp)
• view-source:http://central.maven.org/maven2/archetype-catalog.xml (not applicable: archetype list)
• http://central.maven.org/maven2/org/hibernate/maven-metadata.xml (not applicable: maven plugin list)
• http://central.maven.org/maven2/org/hibernate/hibernate-core/maven-metadata.xml (list of versions per groupId:artifactId)

[FroMage]

The biggest problem would be that Ceylon modules don't have a group/artifact split. We can emulate one, by splitting on the last ., but it may be weird.

[bjansen]

That's already what happens in the pom.xml stored in every car file:

<?xml version="1.0" ?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>ceylon.interop</groupId>
 <artifactId>java</artifactId>
 <version>1.2.2</version>
 <name>ceylon.interop.java</name>
 <dependencies>
  <dependency>
    <groupId>ceylon</groupId>
    <artifactId>collection</artifactId>
    <version>1.2.2</version>
  </dependency>
 </dependencies>
</project>

[bjansen]

Alternatively, we could add an annotation in module.ceylon, something like:

mvn("ceylon", "interop.java")
module ceylon.interop.java 1.0.0 {
    ...
}

[gavinking]

We can emulate one, by splitting on the last ., but it may be weird.

Why can't the group and artifact be identical: the module name?

[luolong]

That's a slippery slope right there -- adding Maven specific annotations to language module...

Maybe have a group annotation instead. This could be an informative tag for Herd, that could be used as a grouping of similar modules together.

And additionally, this would be used for mvn pom generation.

Lacking group annotation, group and artifact id could very well be module's full name

An example:

group("ceylon.sdk")
module ceylon.time 1.2.2 { ...}

Would generate following POM file:

<?xml version="1.0" ?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>ceylon.sdk</groupId>
 <artifactId>ceylon.time</artifactId>
 <name>ceylon.time</name>
 <version>1.2.2</version>
 <dependencies>
  ...
 </dependencies>
</project>

[ckulenkampff]

Usually the groupid refers to the project. See http://central.sonatype.org/pages/choosing-your-coordinates.html

The groupId identifies your project uniquely across all projects and you control this section of the overall name-space.

I think a better default would be:
groupId = Ceylon module name
artifactId = last component of the Ceylon module name

Why can't the group and artifact be identical: the module name?

This might be the best default, because then the artifact would have already the right name for resolution.

[ckulenkampff]

Would you be willing to test this for me?

Yes. I will try to create some kind of integration test for this.

[FroMage]

@vietj: what do you think we should do about group/artifact?

[FroMage]

Note that if we change the group/artifact mapping in Herd, we will also want to change it in the generated pom.xml in the .car files…

[bjansen]

I think Herd should simply extract the pom.xml from the car file, and build the correct hierarchy of folders + generate checksums, that should be enough to expose a Maven repo, right?

[FroMage]

That is indeed another option. Except that:

• some files (jars) may not have any pom.xml
• Herd has more info than the pom.xml files, although cosmetic, such as authors, urls, licence, description, etc…

[davidfestal]

So it should be a mix of both, as we do when generating OSGI metadata in manifest: reuse information specified in the internal POM, and add any additional information that can be provided by Herd.

[bjansen]

reuse information specified in the internal POM, and add any additional information that can be provided by Herd

Looks like a good compromise.

authors, urls, licence, description, etc…

Technically, poms can also contain such data, so we could fill it from module.ceylon if present (is Herd already doing this?).

[FroMage]

I can, that's why I said that it's a bit richer if I generate the pom.xml rather than use the one inside the .car.

[FroMage]

https://modules.ceylon-lang.org/maven/1/ceylon/language/1.2.1/language-1.2.1.pom
https://modules.ceylon-lang.org/maven/1/ceylon/language/maven-metadata.xml

Can you guys try it out?

[FroMage]

Supports .jar, .jar.sha1, -sources.jar, .-sourcesjar.sha1, .pom, .pom.sha1 and maven-metadata.xml.

Does not support javadoc yet but I'm pretty sure the Java tools would not be able to make sense of ceylondoc anyway.

[ckulenkampff]

Wow so fast! I did a small test with Gradle. It works like a charm!

plugins { id 'groovy' }

repositories {
    maven {
        name = 'ceylon-herd'
        url = 'https://modules.ceylon-lang.org/maven/1/'
    }
}


dependencies {
    compile 'ceylon:language:1.2.1'
    compile 'ceylon.interop:java:1.2.1'
    compile 'ceylon:collection:1.2.1'
}

The sources are parsed as Java files, but this should be an Eclipse problem. In Netbeans sources are not shown for Ceylon sources. The Java source of Array.class in ceylon.language is shown, so I think Netbeans looks only for java files as sources :/.

I will try to proxy the Ceylon Herd server with a local Sonatype Nexus server as a second test.

I will have more time tomorrow then I will give you more detailed feedback. Do you think it's useful to have some kind of Gradle/Maven integration tests that can be run against the Herd server?

When I see those artifacts, I really think a fully qualified artifact id would be better.

[FroMage]

Yeah, I think so too, which is why I asked @vietj about his opinion.

[vietj]

having an annotation to specify a group id : can raise a problem if you want to deduce de GAV from the Ceylon module name because you don't have the information

[vietj]

having a scheme with multiple group ids can make a problem later if you want to put the same deps on maven central because usually you owns a single group id

[vietj]

at the same time, maven does not manage cyclic dependencies, so I don't see how you would publish ceylon cyclic dependencies in a maven repo.

[FroMage]

ATM @vietj has parts of the distrib published at http://mvnrepository.com/artifact/org.ceylon-lang under the org.ceylon-lang group.

Using ceylon.language:ceylon.language as coordinates would make publishing to Maven Central harder has every groupId has to be registered (same as on Herd BTW, and we don't see that as a big problem although we do have an issue open to claim domains or wildcards).

[FroMage]

Another option is to use a common groupId for every Ceylon module: ceylon:ceylon.language.

[renatoathaydes]

why not use the same coordinates that already are in maven central?

[renatoathaydes]

Tricky... seems like the only way to avoid trouble is to add a group annotation to the module descriptor... and if the module has no such annotation, it won't be considered for look up in Maven repos?

[renatoathaydes]

instead of a group annotation, maybe the current solution to declare Maven dependencies with a String as in "group:name" "1.0" could be considered as well... for consistency sake.

[ckulenkampff]

Yeah, a decision will severely affect Java-Ceylon-interaction in the long run.

For what purpose is the pom.xml in the CAR generated by Ceylon in the first place?

Maybe the best way to deal with it, is to not generate the pom.xml automatically and to ditch the Maven interoperability in Herd that I proposed here.

All projects that want to be available in Maven would have to add an additional publishing process to their build that generates a pom file and uploads their CAR as JAR to Maven. To make this work you would have to publish Ceylon core stuff also to Maven.

edit: Ah forget it. Then Ceylon projects that use Herd as repository would suffer, because when they use a Java library from Maven that uses a Ceylon library from Maven (which is also used directly by the project with Herd coordinate) CMR would not get that.

[ckulenkampff]

Just for the record, the artifact id is as important as the group id. Both cannot be safely deduced from the Ceylon module name. They are simply two different things. The Ivy Maven interop works so well because Ivy has organization and module name as coordinates which is similar to Mavens coordinates (but also not the same). See http://ant.apache.org/ivy/history/latest-milestone/ivyfile.html / http://ant.apache.org/ivy/history/latest-milestone/ivyfile/info.html

This got complicated very quickly :/ Should we add another issue that deals with Maven / Ceylon artifact coordinate interoperability in another project issue tracker?

[gavinking]

I think what everyone is forgetting here is that the person writing and compiling the module is not the person who wants to depend on the module using mvn. Therefore a system which depends on the module author (optionally) annotating the module at compile time is unlikely to work well in practice.

[bjansen]

Perhaps this has already been discussed in a previous comment, but we're talking about serving a Maven repo from Herd, so why not configure the group id/artifact id in Herd directly? This can be during the project claim, or during the project upload, with default values extracted from the module name.

pros:

• can be easily modified on Herd
• no need to add annotations to module.ceylon
• Herd can generate a new pom, based on all the project settings we fill during the claim/upload
• we don't need to recompile existing modules just to add metadata

cons:

• there's a lot of already existing projects on Herd, so we'll have to add maven coordinates for all of them
• there will be two different pom.xml (the one in the car file, and a second generated by Herd)

[quintesse]

Therefore a system which depends on the module author (optionally) annotating the module at compile time is unlikely to work well in practice.

I don't know why not. I see two situations:

• the author of the module doesn't care about Maven, doesn't (intend to) publish the module to Maven Central so anyone trying to access the module using Maven either has to go through the Herd Maven bridge or has to publish it to some private Maven repo. In both those cases the default group/artifact name we generate would be just fine
• the author does care about Maven and wants to publish it on Maven Central. They do all the work and make sure they annotate their module correctly

We should definitely not support some weird situation where a user of the module decides to publish it on Maven Central!

[quintesse]

@bjansen hmmm not a bad idea at all!

Although honestly I don't like much the fact that 2 (or N, for each time the author decides to change the coordinates in the Herd) different versions of all CARs would exist. That would make checking CRCs a lot harder ("I have 2 sha1s here, one of them should probably check out")

[bjansen]

Do we really need two versions? We can rename the CAR to JAR and not modify the pom.xml inside, Maven will not use anyway (it will use the sibling pom). And I suppose the maven layout will reside in a completely different folder, right?

[bjansen]

Of course this does not solve the problem of uploading things to Maven central, because Maven coordinates won't be known locally. We could add a "publish to maven central" button in Herd, though.

[thradec]

I think, that to have 2 different pom for one car is pretty mess.

2016-03-09 14:40 GMT+01:00 Bastien Jansen notifications@github.com:

Do we really need two versions? We can rename the CAR to JAR and not
modify the pom.xml inside, Maven will not use anyway (it will use the
sibling pom). And I suppose the maven layout will reside in a completely
different folder, right?

—
Reply to this email directly or view it on GitHub
#262 (comment).

[bjansen]

Can someone please answer this?

For what purpose is the pom.xml in the CAR generated by Ceylon in the first place?

[quintesse]

Maven will not use anyway

If that's true then okay. It might be weird to see that Ceylon's metamodel would return different names than the one you asked for but we could probably live with that.

[quintesse]

Can someone please answer this?

As far as I know because that's what many (all?) Maven artifacts do too, so I guess it's used somewhere. But I think @FroMage will know better

[gavinking]

we're talking about serving a Maven repo from Herd, so why not configure the group id/artifact id in Herd directly?

Right, that's exactly what I was getting at in my last comment. Stef has already decided to generate a pom using Herd itself (rather than just extracting it from the car). So it doesn't seem unreasonable to have Herd handle this stuff as well.

[FroMage]

That's possible, but it does not make it easier for people to upload to Maven Central without Herd in the equation.

[gavinking]

It's not really clear to me how and when and why the "upload to Maven Central" requirement snuck into this particular issue. Originally we were talking about letting Herd act as a mvn repo.

[ckulenkampff]

Maybe it helps to list use cases and possible problems:

1. As a Java developer, I want to develop a Java Gradle project and want to use dependencies only available on Herd.

Currently, I can add the Herd repository as Maven repository in Gradle. This works fine for now.

2. As a Java developer, I want to develop a Java Gradle project that uses dependencies only available on Herd and some Maven Central dependencies.

Currently, I can add both, a Maven repository and the Herd repository, to my Gradle project. This works fine for now.

3. As a Java developer, I want to develop a Java Gradle project that uses a Ceylon library only available at Herd with a transitive dependency to com.fasterxml.jackson.core. My project also uses a big Java library which also depends on the Maven variant com.fasterxml.jackson.core:jackson-core.

Currently, I can add both, a Maven repository and the Herd repository, to my Gradle project. But the jackson-core dependency will be downloaded two times with different names and pollutes my classpath.

There are similar use cases for Ceylon developers where different names should result in multiple downloads and imports of the same library (even if Maven fake mode of Herd is ditched, at least when eclipse-archived/ceylon#5968 is solved).

One can always resolve these issues manually, but it makes Ceylon Herd look like an offender.

The only solution to this is to somehow link dependencies on Maven Central with dependencies on Herd (when they are uploaded to both). When maintaining this link is manual work, many people will not do it, which in turn results in ugly problems mentioned in 3). One solution would be to allow Herd to push my library to Maven Central or another popular Maven repository. This way the link will be automatically maintained by Herd.

This seems also to be what JCenter makes possible for its users:

And if you're into legacy, you can even synchronize your packages directly to Maven Central.

BTW similar issues might occur when working on NPM interoperability, but I am not sure...

[ckulenkampff]

Just an update: Tested it with a local Sonatype Nexus installation. I could add the Herd repository as proxy.

I think the following configuration was necessary to make it work better:

[luolong]

Now, there's been so much discussion here hat I might just have skimmed over some relevant bits. If so, excuse me if I rehash something that has already been decided.

About my suggestion for setting Maven groupId:artifactId identifiers pair --
The way I see this is that we have to have a sensible default that does not make people surprised if they do the default thing.

Then we should have some way of overriding these defaults. First on the command line and then for perpetuating the chosen setting in a configuration file somewhere (.ceylon/config seems good place)

The defaults

The most sensible default behavior I can come up with is to have group id and artifact id to be same as module id.

I know that this runs counter to how Ceylon modules have been published to maven so far, but if this is configurable later, I still think that this is the most sensible default.

Command line

One should be able to configure maven groupId and artifactId on the command line. In its simplest form, we could just set those up as plain text values. If a project has just one module, this should be simple enough:

ceylon compile --maven-groupId=ceylon-lang

The above command should produce set of modules whose groupId is set to ceylon-lang and artifactId is full module name

ceylon compile --maven-groupid=ceylon-lang --maven-artifactid=interop-java ceylon.interop-java

would produce ceylon module with maven coordinates 'ceylon-lang:interop-java:1.2.2'

Trying to call the latter command line with multiple modules (either explicitly on command line, or implicitly when calling without module name in a project with multiple modules) should be considered an error.

As an additional consideration, we could add support for some simplified regular expression syntax for deriving groupId and artifactId from the module name. Something along these lines:

ceylon compile --maven-groupid=ceylon-lang --maven-artifactId="ceylon.(**)|$1|r/./-"

That would take everything after "ceylon." in the module name and use it as Maven artifactId ...
The exact syntax of the regular expression is kind of irrelevant at this point, but it should be simple enough to be able to do limited set of operations on the initial input string (module name):

• Select a portion of a string (preferably being aware of dots separating segments of module name)
• replace one string (or character) with another (e.g. replacing dots with dashes)

Project configuration

Whatever we allow on command line we should be able to configure in .ceylon/config file. With additional freedom of being able to override settings on per-module basis.

I guess the simplest configuration might be like this:

[defaults]
encoding=UTF-8
overrides=build/overrides.xml

maven-groupid=ceylon-lang
maven-artifactId=ceylon.(**)|$1
...

[module "ceylon.interop.java"]
maven-artifactId=interop-java

[vietj]

this seems to be a great debate, however I haven't see anything about cyclic dependencies.

how are cyclic module dependencies going to be exposed by Herd to Maven ?

[luolong]

I guess we have several layers of maven/ivy interop issues under discussion here ... maybe we should split those discussions off to their own separate issues?