Port most of the build to gradle. #343

marshallpierce · 2019-05-26T16:54:41Z

This is still missing some stuff but I wanted to get feedback on if this was useful / interesting before sinking more time into it.

Typical gradle subproject structure plus:

Multi-release jar for shims subproject
Move examples into typical src/main/java and set up tasks for each one, removing the need for a separate shell script
Move JMH benchmarks from src/main/java to src/jmh/java since that's how the JMH plugin prefers it. (This allows benchmarks to live in the same module as the normal code, if desired.)

Missing things:

Deploying releases (use Bintray to avoid Sonatype Nexus pain?)
jxr
perhaps other subtleties I missed/forgot

This is still missing some stuff but I wanted to get feedback on if this was useful / interesting before sinking more time into it. Typical gradle subproject structure plus: - Multi-release jar for shims subproject - Move examples into typical `src/main/java` and set up tasks for each one, removing the need for a separate shell script - Move JMH benchmarks from `src/main/java` to `src/jmh/java` since that's how the JMH plugin prefers it. (This allows benchmarks to live in the same module as the normal code, if desired.) Missing things: - Deploying releases (use Bintray to avoid Sonatype Nexus pain?) - jxr - perhaps other subtleties I missed/forgot

richardstartin · 2019-05-26T19:13:24Z

This looks very promising.

Regarding the multi release functionality, benchmarks still needs to be multi release so the right implementation gets loaded (maven-shade took care of this).

The reason for targeting JDK11+ in shims was to simplify testing, as the functionality was introduced after JDK10 was officially retired. Since we couldn't build with JDK8 any more, the idea was that building with JDK10 would exercise the JDK8 implementation. There are probably other ways to achieve this effect.

marshallpierce · 2019-05-26T19:16:39Z

Ah, k. We can drop in the shadow plugin to do more fancy uberjaring than the jmh plugin does out of the box.

I imagine most people who are on Java > 8 and care about performance are on Java 11 by this point anyway, so while we could get fancy to enable multi-release shim for Java 9, that seems like rapidly diminishing ROI.

lemire · 2019-05-26T19:20:17Z

JDK10 may die soon.

marshallpierce · 2019-05-26T19:20:58Z

Both 9 and 10 are not supported long term IIRC... 10 may continue to serve as "not java 8 but not java 11" for some time I suspect :)

richardstartin · 2019-05-26T19:23:01Z

That's exactly what it's doing. The only question is how long Travis will make JDK10 available. By that time JDK11 may have more widespread adoption.

richardstartin · 2019-05-26T19:28:03Z

Multi release jars may not be the best solution for using new features while supporting old ones. Multiple build targets may be a better option.

marshallpierce · 2019-05-26T19:30:04Z

Cedric has thoughts on the matter: https://blog.gradle.org/mrjars. Haven't tangled with them before personally.

richardstartin · 2019-05-26T20:09:12Z

I just checked out your branch and ran ./gradlew build and the fuzz tests ran. Currently these are enabled only by a profile because it takes hours to run them. We'll need an environment variable to enable them.

marshallpierce · 2019-05-27T00:32:39Z

If this is looking good so far, the next step would be to set up deploying released artifacts.

I suggest switching to Bintray:

I haven't had any issues with certain artifacts out of the group for a particular release going missing
When you publish a new version, it "stages" those files so you can click around in the web ui and make sure all the files you expect are there before it really publishes them
It will GPG sign for you so you don't have to deal with that, though you can still do it yourself if you're fond of GPG for some reason
It will sync your Bintray project to jcenter, which is a superset of maven central
And it's free for open source use

If you're OK with that, I can drop in the setup I have here: https://bitbucket.org/marshallpierce/gcp-logging-enhancers/src/master/build.gradle. My workflow is ./gradlew release. It asks me for the new version number, does the various alterations/commits/tags, uploads artifacts to bintray, and sets the next version number snapshot.

lemire · 2019-05-27T01:21:06Z

I am fine with switching to a new release setup.

For the moment publishing only works to maven local.

marshallpierce · 2019-05-27T13:47:36Z

I've created a bintray org. Let me know your usernames and I can add you.

marshallpierce · 2019-05-27T14:14:16Z

I let it try to release a 0.0 version into Bintray (we can delete the version once we're done testing).

I did: ./gradlew release with the version temporarily set to 0.0-SNAPSHOT and other settings altered to allow me to run it with local changes and not on the master branch:

Output (with think time for me to press enter at the right places):

> Task :RoaringBitmap:checkCommitNeeded
!!WARNING!! You have uncommitted files:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 M build.gradle.kts
 M gradle.properties
AM release.gradle
R  pom.xml -> zpom.xml
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

> Task :RoaringBitmap:confirmReleaseVersion

??> This release version: [0.0]  (WAITING FOR INPUT BELOW)
<=====--------> 40% EXECUTING [4s]

> Task :RoaringBitmap:RoaringBitmap:shims:bintrayUpload
Uploading to https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/shims/0.0/shims-0.0-sources.jar...
Uploaded to 'https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/shims/0.0/shims-0.0-sources.jar'.
Uploading to https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/shims/0.0/shims-0.0-javadoc.jar...
Uploaded to 'https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/shims/0.0/shims-0.0-javadoc.jar'.
Uploading to https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/shims/0.0/shims-0.0.jar...
Uploaded to 'https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/shims/0.0/shims-0.0.jar'.
Uploading to https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/shims/0.0/shims-0.0.pom...
Uploaded to 'https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/shims/0.0/shims-0.0.pom'.

> Task :RoaringBitmap:RoaringBitmap:roaringbitmap:bintrayUpload
Uploading to https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/roaringbitmap/0.0/roaringbitmap-0.0-sources.jar...
Uploaded to 'https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/roaringbitmap/0.0/roaringbitmap-0.0-sources.jar'.
Uploading to https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/roaringbitmap/0.0/roaringbitmap-0.0-javadoc.jar...
Uploaded to 'https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/roaringbitmap/0.0/roaringbitmap-0.0-javadoc.jar'.
Uploading to https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/roaringbitmap/0.0/roaringbitmap-0.0.jar...
Uploaded to 'https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/roaringbitmap/0.0/roaringbitmap-0.0.jar'.
Uploading to https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/roaringbitmap/0.0/roaringbitmap-0.0.pom...
Uploaded to 'https://api.bintray.com/content/roaringbitmap/maven/RoaringBitmap/0.0/org/roaringbitmap/roaringbitmap/0.0/roaringbitmap-0.0.pom'.

> Task :RoaringBitmap:updateVersion

??> Enter the next version (current one released as [0.0]): [0.1-SNAPSHOT]  (WAITING FOR INPUT BELOW)
<============-> 93% EXECUTING [35s]

BUILD SUCCESSFUL in 38s
1 actionable task: 1 executed

In other words, it figured out that 0.0-SNAPSHOT should be released as 0.0, created and uploaded jars, and set the next version as 0.1-SNAPSHOT. The jars are unpublished but waiting in https://bintray.com/roaringbitmap/maven/RoaringBitmap. Once you're in the organization you should be able to see them and make sure things look ok.

richardstartin · 2019-05-27T14:46:22Z

I have some reservations about publishing to jcenter via bintray - it’s usually impossible to pull in dependencies from jcenter within more security conscious organisations. Maven central is more widely trusted.

marshallpierce · 2019-05-27T14:50:05Z

If you give bintray your maven central credentials it can auto-push there.

richardstartin · 2019-05-29T18:23:17Z

OK that sounds very reasonable to me.

marshallpierce · 2019-06-20T18:23:05Z

OK, to make sure I'm following:

./gradlew release correctly uploads stuff to Bintray, which also signs them
Clicking publish in bintray (you should be able to click just once, from the level of the maven repo, and have it take effect for both packages) confirms that you do want to keep those artifacts.
- I agree it is a second step which is clumsy, but I've found it valuable to be able to give the artifacts a once-over before I publish them. Maybe we can turn that off if you prefer?
But then pushing them to maven central fails because of inscrutable nexus problems?

lemire · 2019-06-20T18:32:42Z

./gradlew release correctly uploads stuff to Bintray, which also signs them

I cannot sign with my keys, but I apparently can sign with bintray's keys.

Or rather, I could sign with my keys, but that would require me writing a REST app.

Clicking publish in bintray confirms that you do want to keep those artifacts. I agree it is a second step which is clumsy, but I've found it valuable to be able to give the artifacts a once-over before I publish them.

You mean that at this stage, you run additional checks? What kind of checks?

(you should be able to click just once, from the level of the maven repo, and have it take effect for both packages)

I don't know how to do so. It seems that I have to go into the package, the exact version number, then click on maven central then click on sync... and I have to do with each "sub" package individually.

There is no "do everything" button. I have to click, click, click... that's what I mean by the fact that it is not automated.

But then pushing them to maven central fails because of inscrutable nexus problems?

Correct.

lemire · 2019-06-20T18:38:51Z

Ok. Now a new error...

Last Sync Errors: Missing MD5: '/org/roaringbitmap/shims/0.8.9/shims-0.8.9-sources.jar.md5' Missing Signature: '/org/roaringbitmap/shims/0.8.9/shims-0.8.9-sources.jar.asc' does not exist for 'shims-0.8.9-sources.jar'. Dropping existing partial staging repository.

That's when trying to sync RoaringBitmap (and not shims).

marshallpierce · 2019-06-20T18:42:36Z

You mean that at this stage, you run additional checks? What kind of checks?

I'll look around at the upload files for the new version and see that they look right (.jar, -sources.jar, etc). Nothing too heavy.

I haven't gone this far down the path of trying to appease Nexus, but I can take a look at generating .md5s of all jars automatically. (MD5? really?)

lemire · 2019-06-20T18:43:08Z

New error (after trying again):

Last Sync Errors: Missing: no sources jar found in folder '/org/roaringbitmap/shims/0.8.9' Dropping existing partial staging repository.

marshallpierce · 2019-06-20T18:44:21Z

That's weird... Sure looks like there's a sources jar: https://bintray.com/roaringbitmap/maven/org.roaringbitmap%3Ashims/0.8.9#files/org%2Froaringbitmap%2Fshims%2F0.8.9

lemire · 2019-06-20T18:45:04Z

@marshallpierce

Presumably, you have access to the page, right...?

https://bintray.com/roaringbitmap/maven/org.roaringbitmap%3ARoaringBitmap#central

lemire · 2019-06-20T18:46:43Z

My guess is that I can keep going all day, getting random errors one after the other. There is probably a fun script at the other end with a random message generator.

marshallpierce · 2019-06-20T18:46:52Z

Yep, I can see that page. Maybe hand off to bintray support again? The sources jar is definitely there.

marshallpierce · 2019-06-20T18:47:27Z

This kind of stuff is why I gave up on syncing to maven central for my own projects. :/

lemire · 2019-06-20T18:48:23Z

@marshallpierce Well... is it though? Because that's not the shim package, that's the RoaringBitmap package.

Normally, in the maven release cycle, we upload everything in one go. There is one unified release. Now we have to proceed in two steps, but the steps are interdependent.

lemire · 2019-06-20T18:51:55Z

@marshallpierce ... because the message comes from maven central. It is maven central saying that we are missing shims sources.

marshallpierce · 2019-06-20T18:52:39Z

Bintray has the sources jar for RB as well: https://bintray.com/roaringbitmap/maven/org.roaringbitmap%3ARoaringBitmap#files/org%2Froaringbitmap%2FRoaringBitmap%2F0.8.9

lemire · 2019-06-20T18:53:25Z

Ok. I'll open a new issue.

lemire · 2019-06-20T18:56:15Z

Please see #345

lemire · 2019-06-20T20:11:06Z

@marshallpierce BTW the shims package is now on maven...
https://repo1.maven.org/maven2/org/roaringbitmap/shims/0.8.9/

lemire · 2019-06-20T20:46:15Z

Success... after trying all afternoon, everything is on maven, finally.

@marshallpierce Can you merge the PR I have issued against your PR... ? Then we will merge your PR.

We know that if we are very patient, we can issue a working release from bintray to maven central. And I have checked that it is functional.

I guess that this is needed to keep things in sync.

lemire · 2019-06-20T20:57:36Z

@marshallpierce Can you merge once more? I have updated the docs.

I have already sent a Roaring Bitmap sticker to @richardstartin , if you want one, just share your mailing address with me...

blacelle · 2019-06-20T21:05:12Z

Happy success guys !

lemire · 2019-06-20T21:08:29Z

@blacelle Want a Roaring sticker? https://twitter.com/lemire/status/1139649200222855169

Updating instructions...

lemire · 2019-06-20T21:12:08Z

Let us wait to make sure that the tests pass, and then we can merge...

This must have been the hardest PR to merge, ever.

lemire · 2019-06-20T21:14:43Z

Tests are failing. Restarting.

lemire · 2019-06-20T21:33:34Z

Tests are passing. Merging.

* Port most of the build to gradle. This is still missing some stuff but I wanted to get feedback on if this was useful / interesting before sinking more time into it. Typical gradle subproject structure plus: - Multi-release jar for shims subproject - Move examples into typical `src/main/java` and set up tasks for each one, removing the need for a separate shell script - Move JMH benchmarks from `src/main/java` to `src/jmh/java` since that's how the JMH plugin prefers it. (This allows benchmarks to live in the same module as the normal code, if desired.)

marshallpierce force-pushed the gradle branch 2 times, most recently from 7327900 to 725094b Compare May 26, 2019 17:02

marshallpierce force-pushed the gradle branch from 725094b to 483d7c9 Compare May 26, 2019 17:47

marshallpierce added 2 commits May 26, 2019 12:17

Use JUnit temp folder rule isntead of assuming target dir exists

08692c0

Target JDK9 when compiling the shim, but only use it on 11+.

dadbad7

Include jmh benchmarks in initial Travis compilation.

aed33e8

marshallpierce added 3 commits May 26, 2019 17:32

Disable fuzz tests unless roaringbitmap.fuzz-tests prop is set

29979ef

Make jmhJar a dependency of assemble.

bdd54c4

Set up shadow jar for jmh benchmarks

3567661

marshallpierce added 2 commits May 27, 2019 06:54

Tweak shims build to include shims in all cases

8f15aaa

Add maven publications and release plugin.

53a415f

For the moment publishing only works to maven local.

Add bintray config

43e7550

marshallpierce force-pushed the gradle branch from 035e226 to 43e7550 Compare May 27, 2019 14:09

marshallpierce mentioned this pull request May 29, 2019

Plugin 0.4.2 is broken: jar without dependencies melix/jmh-gradle-plugin#97

Open

lemire mentioned this pull request Jun 20, 2019

Dropping support for Maven Central and bintray #345

Closed

marshallpierce and others added 2 commits June 20, 2019 14:50

Merge pull request #1 from lemire/master

2ab9f7c

I guess that this is needed to keep things in sync.

Updating instructions...

3866a05

lemire and others added 2 commits June 20, 2019 17:10

Correcting.

f1c507c

Merge pull request #2 from lemire/master

be82739

Updating instructions...

lemire merged commit bda3d06 into RoaringBitmap:master Jun 20, 2019

Port most of the build to gradle. #343

Port most of the build to gradle. #343

Conversation

marshallpierce commented May 26, 2019

richardstartin commented May 26, 2019

marshallpierce commented May 26, 2019

lemire commented May 26, 2019

marshallpierce commented May 26, 2019

richardstartin commented May 26, 2019

richardstartin commented May 26, 2019

marshallpierce commented May 26, 2019

richardstartin commented May 26, 2019

marshallpierce commented May 27, 2019

lemire commented May 27, 2019

marshallpierce commented May 27, 2019

marshallpierce commented May 27, 2019

richardstartin commented May 27, 2019

marshallpierce commented May 27, 2019

richardstartin commented May 29, 2019

marshallpierce commented Jun 20, 2019

lemire commented Jun 20, 2019

lemire commented Jun 20, 2019

marshallpierce commented Jun 20, 2019

lemire commented Jun 20, 2019

marshallpierce commented Jun 20, 2019

lemire commented Jun 20, 2019

lemire commented Jun 20, 2019

marshallpierce commented Jun 20, 2019

marshallpierce commented Jun 20, 2019

lemire commented Jun 20, 2019

lemire commented Jun 20, 2019

marshallpierce commented Jun 20, 2019

lemire commented Jun 20, 2019

lemire commented Jun 20, 2019

lemire commented Jun 20, 2019

lemire commented Jun 20, 2019

lemire commented Jun 20, 2019

blacelle commented Jun 20, 2019

lemire commented Jun 20, 2019

lemire commented Jun 20, 2019

lemire commented Jun 20, 2019

lemire commented Jun 20, 2019