Make the release process compatible with the Apache guidelines

[bchapuis]

We should use the apache infrastructure to publish a first incubating release.

The releases are currently being published automatically on maven central (jar files), docker hub (image generated with jib) and github (zip distribution) and we should adapt the workflow. The issues #219 and #184 may also be relevant here. JReleaser looks really nice and may help at simplifying the release work.

The following checklists aims at listing the changes that must be performed in the workflow:

Maven

• publish artifacts to the maven central repository
• artifacts should be under the groupId of org.apache.baremaps
• The POM file should set the ALv2 license as its license, Apache Baremaps as its developer name, and proper source control information.
• the project description should include the incubator disclaimer
• Release candidates, nightlys or snapshots need to be clearly marked by adding suffix in its version, like org.apache.baremaps.:0.1-SNAPSHOT
• publish the artifacts to the apache maven repository #719
• Release automation #844

Github

• publish binary release github (baremaps/baremaps)
• publish binary release github (apache/incubator-baremaps)
• any releases need to include the text of the incubation disclaimer.
• the release page must not contain release candidates, nightly or snapshots releases that have not been tagged as prereleases (Ignoring that GitHub also displays tags on the release page)
• Any releases that exist before coming into incubation need to be clearly described on the release page and tagged as such on https://github.com/apache/incubator-baremaps/tags
• release candidates, nightlys or snapshots releases can be tagged and appear on https://github.com/apache/incubator-baremaps/tags.

Legal

• Verify the copyright and attribution of external code included in the project
• Verify the licenses of the dependencies
• Check the project with Skywalking #720
• Configure Maven Rat #731

Misc

• include the word incubating in the release file name
• automate the creation of a changelog file
• ensure that the copyrights is present in the binary distribution
• include a NOTICE file in the binary distribution
• include a LICENSE file in the binary distribution
• have valid checksums
• have valid signatures
• the official release artifacts should not contain unapproved code
• have a KEYS file to validate the release
• Audit the various test data sets #721
• Check any generated files in the source code? maplibre-gl-inspect.js and schema.db
• Add headers to every text file that supports it, e.g. pom.xml, README.md, baremaps.bat (Fix missing license headers #598)
• Check the licenses with the maven rat plugin #603
• Check the licenses with the maven rat plugin #603
• Missing files in src release #604
• Add prefix apache- on the tar file names and on the directory inside the tar #605
• Add version number in README in the release #606
• Verify the projects necessary in the NOTICE file #607
• Verify and correct the LICENSE file for the -bin.tar release #608

Sources

• https://incubator.apache.org/guides/releasemanagement.html
• https://incubator.apache.org/guides/distribution.html
• https://apache.org/legal/resolved.html
• https://infra.apache.org/licensing-howto.html#bundle-asf-product
• https://infra.apache.org/publishing-maven-artifacts.html
• https://infra.apache.org/licensing-howto.html#binary

[bchapuis]

After playing with jreleaser, I think we should keep the maven release plugin for now. It looks like the plugin is not able to release sources.

#531 configures JReleaser. The release.sh script give an overview of the process. An early-access tag is automatically created and allows to preview the release.

[bchapuis]

@CalvinKirs @julianhyde @LeonardBesseau I'm progressing on the various tasks associated with the first release, and would like to have your opinion on PGP signatures.

Right now, we do have an action that signs the jar files published on maven central. A PGP key dedicated to the project is stored as a secret on github. Do you think it fine to use a similar approach to sign the source and binary distributions published on github? We may ask the committers to sign the project's key, but I'm not totally sure about the implications of this.

This approach would make the release process very simple. In order to release, one of the committer would have to execute the following:

mvn release:prepare -DautoVersionSubmodules=true -DgenerateBackupPoms=false

As the release plugin creates a tag for the release (vX.Y.Z), the creation of the assets can be automated with an action. Furthermore, the release can remain a draft until the vote passes on the mailing list.

[CalvinKirs]

@CalvinKirs @julianhyde @LeonardBesseau I'm progressing on the various tasks associated with the first release, and would like to have your opinion on PGP signatures.

Right now, we do have an action that signs the jar files published on maven central. A PGP key dedicated to the project is stored as a secret on github. Do you think it fine to use a similar approach to sign the source and binary distributions published on github? We may ask the committers to sign the project's key, but I'm not totally sure about the implications of this.

This approach would make the release process very simple. In order to release, one of the committer would have to execute the following:

mvn release:prepare -DautoVersionSubmodules=true -DgenerateBackupPoms=false

As the release plugin creates a tag for the release (vX.Y.Z), the creation of the assets can be automated with an action. Furthermore, the release can remain a draft until the vote passes on the mailing list.

TBH, I haven't done that.

We usually do this when we package the source code for a release, because the source code package also needs to sign.

At the same time, the release is more done on the computer of the Release Manager, because there is some information such as security key. we can't all share a key.

Usually most projects will have corresponding release scripts, which can help release versions quickly,
https://github.com/apache/skywalking/blob/master/tools/releasing/create_source_release.sh

I'm guessing you missed this documentation
https://infra.apache.org/publishing-maven-artifacts.html

[CalvinKirs]

This is all the process of general release:
https://skywalking.apache.org/docs/main/v9.2.0/en/guides/how-to-release/

[bchapuis]

Thank you for the pointers.

I'd love the source and binary releases to be generated by the CI to prevent issues such as differences in newline encoding. A possibility may be to have an addhoc script to sign the releases produced by the CI on the computer of the release manager. I will look at your links and try to figure this out.

[aalmiray]

After playing with jreleaser, I think we should keep the maven release plugin for now. It looks like the plugin is not able to release sources.

Hello there! I'm the JReleaser author. The tool can post any files as release assets, not just binaries. If your goal is to post source archives then they may be added as artifacts under the <files> (https://jreleaser.org/guide/latest/reference/files.html) section, not <distributions>.

FWIW the mvnd project accepted a PR to use JReleaser in such a way that it would be compatible with the ASF's rules. However, it was deactivated after merging because the project requires explicit permission to use an external GH action -> https://issues.apache.org/jira/browse/INFRA-23128 Your case may be different if you only use the jreleaser-maven-plugin.

[bchapuis]

@aalmiray Thanks a lot for reaching out and for your work on jreleaser. I'm getting familiar with the Apache guidelines and from what I understand the most important part of a release is the source code. The binaries, artifacts, etc. are provided as a convenient way to install the project, but they are not really considered as being part of the release.

Regarding your comment, from what I understand, I should use the maven assembly plugin to generate an archive containing the source code, and then use JReleaser and the <files> section to upload this artifact. Is that correct? What I was originally looking for was a way to do this with JReleaser.

[aalmiray]

Regarding your comment, from what I understand, I should use the maven assembly plugin to generate an archive containing the source code, and then use JReleaser and the <files> section to upload this artifact. Is that correct? What I was originally looking for was a way to do this with JReleaser.

That is correct. The maven assembly plugin would be my first recommendation if you're familiar with it. JReleaser can also create archives via its archive assembler. v1.5.0 which comes out at the end of February 2023 (in just a few days) adds finer control to define which files get added to an archive.

If this assembler is used I'd recommend setting its exported property to false otherwise you'll get a matching distribution of type BINARY which could inadvertently be used in combination with package managers. If you do not configure any package managers (brew, docker, etc) then this shouldn't be a problem. Setting exported to false and using the files section would be the way to go for the time being until JReleaser adds proper support for distributions of type SOURCE (see related discussion topic).

[bchapuis]

This clarifies things a lot. In the context of baremaps, we really need to provide a binary distribution and a docker image in addition to maven artifacts. This issue is regularly reported by users who don't want to build from sources. I will have take some time to investigate the use of JReleaser again.

[aalmiray]

Great! Let me know if you have any questions 😄

[joshfischer1108]

Hi,

Do we have any of the work accounted for in terms of listing out Baremaps' dependency licenses?

[bchapuis]

Hi @joshfischer1108, the java dependencies have been carefully reviewed. The remaining issues are related to: a dataset located in proj4j-epsg (an upstream contribution is possible and needed); some datasets used in the tests; minified javascript libraries and fonts included in the resources directory (not sure if including these resources from a CDN would be sufficient). Regarding the listing are we supposed to include something in the release?

Here is the list of third-party licenses (mvn license:add-third-party). Notice that JSQLParser uses a dual licence (APACHEV2 and LGPLV21).

Lists of 81 third-party dependencies.
     (The Apache Software License, Version 2.0) Jackson-annotations (com.fasterxml.jackson.core:jackson-annotations:2.13.0 - http://github.com/FasterXML/jackson)
     (The Apache Software License, Version 2.0) Jackson-core (com.fasterxml.jackson.core:jackson-core:2.13.0 - https://github.com/FasterXML/jackson-core)
     (The Apache Software License, Version 2.0) jackson-databind (com.fasterxml.jackson.core:jackson-databind:2.13.0 - http://github.com/FasterXML/jackson)
     (The Apache Software License, Version 2.0) Jackson-dataformat-CSV (com.fasterxml.jackson.dataformat:jackson-dataformat-csv:2.13.0 - https://github.com/FasterXML/jackson-dataformats-text)
     (The Apache Software License, Version 2.0) Jackson-dataformat-YAML (com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.0 - https://github.com/FasterXML/jackson-dataformats-text)
     (The Apache Software License, Version 2.0) Jackson datatype: jdk8 (com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.13.0 - https://github.com/FasterXML/jackson-modules-java8/jackson-datatype-jdk8)
     (Apache License, Version 2.0) Caffeine cache (com.github.ben-manes.caffeine:caffeine:3.1.1 - https://github.com/ben-manes/caffeine)
     (The Apache Software License, Version 2.0) docker-java-api (com.github.docker-java:docker-java-api:3.2.13 - https://github.com/docker-java/docker-java)
     (The Apache Software License, Version 2.0) docker-java-transport (com.github.docker-java:docker-java-transport:3.2.13 - https://github.com/docker-java/docker-java)
     (The Apache Software License, Version 2.0) docker-java-transport-zerodep (com.github.docker-java:docker-java-transport-zerodep:3.2.13 - https://github.com/docker-java/docker-java)
     (GNU Library or Lesser General Public License (LGPL) V2.1) (The Apache Software License, Version 2.0) JSQLParser library (com.github.jsqlparser:jsqlparser:4.2 - https://github.com/JSQLParser/JSqlParser)
     (The Apache Software License, Version 2.0) FindBugs-jsr305 (com.google.code.findbugs:jsr305:3.0.2 - http://findbugs.sourceforge.net/)
     (Apache 2.0) error-prone annotations (com.google.errorprone:error_prone_annotations:2.14.0 - https://errorprone.info/error_prone_annotations)
     (Apache License V2.0) FlatBuffers Java API (com.google.flatbuffers:flatbuffers-java:22.10.26 - https://github.com/google/flatbuffers)
     (The Apache Software License, Version 2.0) Guava InternalFutureFailureAccess and InternalFutures (com.google.guava:failureaccess:1.0.1 - https://github.com/google/guava/failureaccess)
     (Apache License, Version 2.0) Guava: Google Core Libraries for Java (com.google.guava:guava:31.1-jre - https://github.com/google/guava)
     (The Apache Software License, Version 2.0) Guava ListenableFuture only (com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava - https://github.com/google/guava/listenablefuture)
     (The Apache Software License, Version 2.0) J2ObjC Annotations (com.google.j2objc:j2objc-annotations:1.3 - https://github.com/google/j2objc/)
     (BSD-3-Clause) Protocol Buffers [Core] (com.google.protobuf:protobuf-java:3.21.6 - https://developers.google.com/protocol-buffers/protobuf-java/)
     (ISC License) ORMLite JDBC (com.j256.ormlite:ormlite-jdbc:6.1 - https://ormlite.com/)
     (The Apache Software License, Version 2.0) HikariCP (com.zaxxer:HikariCP:5.0.1 - https://github.com/brettwooldridge/HikariCP)
     (MIT License) pgbulkinsert (de.bytefish:pgbulkinsert:8.1.0 - http://www.github.com/bytefish/PgBulkInsert)
     (The Apache Software License, Version 2.0) Google S2 geometry library (io.sgr:s2-geometry-library-java:1.0.0 - https://github.com/sgr-io/s2-geometry-library-java)
     (Apache License, Version 2.0) fastutil (it.unimi.dsi:fastutil:8.5.9 - http://fastutil.di.unimi.it/)
     (Eclipse Public License 1.0) JUnit (junit:junit:4.13.2 - http://junit.org)
     (The MIT License (MIT)) Color (mil.nga:color:1.0.0 - https://github.com/ngageoint/color-java)
     (The MIT License (MIT)) Coordinate Reference Systems (mil.nga:crs:1.1.4 - https://github.com/ngageoint/coordinate-reference-systems-java)
     (The MIT License (MIT)) Projections (mil.nga:proj:1.1.2 - https://github.com/ngageoint/projections-java)
     (The MIT License (MIT)) Simple Features (mil.nga:sf:2.2.0 - https://github.com/ngageoint/simple-features-java)
     (The MIT License (MIT)) Tagged Image File Format (mil.nga:tiff:3.0.0 - https://github.com/ngageoint/tiff-java)
     (The MIT License (MIT)) GeoPackage (mil.nga.geopackage:geopackage:6.5.0 - https://github.com/ngageoint/geopackage-java)
     (The MIT License (MIT)) GeoPackage Core (mil.nga.geopackage:geopackage-core:6.5.0 - https://github.com/ngageoint/geopackage-core-java)
     (The MIT License (MIT)) OGC API Features JSON (mil.nga.oapi.features:oapi-features-json:2.2.0 - https://github.com/ngageoint/ogc-api-features-json-java)
     (The MIT License (MIT)) Simple Features GeoJSON (mil.nga.sf:sf-geojson:3.2.0 - https://github.com/ngageoint/simple-features-geojson-java)
     (The MIT License (MIT)) Simple Features Projection (mil.nga.sf:sf-proj:4.3.0 - https://github.com/ngageoint/simple-features-proj-java)
     (The MIT License (MIT)) Simple Features Well-Known Binary (mil.nga.sf:sf-wkb:2.2.0 - https://github.com/ngageoint/simple-features-wkb-java)
     (The MIT License (MIT)) Simple Features Well-Known Text (mil.nga.sf:sf-wkt:1.2.0 - https://github.com/ngageoint/simple-features-wkt-java)
     (Apache License v2.0) (LGPL, version 2.1) Java Native Access (net.java.dev.jna:jna:5.8.0 - https://github.com/java-native-access/jna)
     (The BSD License) RIPE NCC - IP Resource (net.ripe.ipresource:ipresource:1.49 - https://github.com/RIPE-NCC/ipresource)
     (Apache License, Version 2.0) Apache Commons Compress (org.apache.commons:commons-compress:1.21 - https://commons.apache.org/proper/commons-compress/)
     (The Apache Software License, Version 2.0) Commons Lang (org.apache.commons:commons-lang3:3.1 - http://commons.apache.org/lang/)
     (Apache License, Version 2.0) Apache Log4j API (org.apache.logging.log4j:log4j-api:2.19.0 - https://logging.apache.org/log4j/2.x/log4j-api/)
     (Apache License, Version 2.0) Apache Log4j Core (org.apache.logging.log4j:log4j-core:2.19.0 - https://logging.apache.org/log4j/2.x/log4j-core/)
     (Apache License, Version 2.0) Apache Log4j SLF4J 2.0 Binding (org.apache.logging.log4j:log4j-slf4j2-impl:2.19.0 - https://logging.apache.org/log4j/2.x/log4j-slf4j2-impl/)
     (Apache 2) Apache Lucene (module: core) (org.apache.lucene:lucene-core:9.4.2 - https://lucene.apache.org/)
     (Apache 2) Apache Lucene (module: queries) (org.apache.lucene:lucene-queries:9.4.2 - https://lucene.apache.org/)
     (Apache 2) Apache Lucene (module: queryparser) (org.apache.lucene:lucene-queryparser:9.4.2 - https://lucene.apache.org/)
     (Apache 2) Apache Lucene (module: sandbox) (org.apache.lucene:lucene-sandbox:9.4.2 - https://lucene.apache.org/)
     (Apache 2) Apache Lucene (module: spatial-extras) (org.apache.lucene:lucene-spatial-extras:9.4.2 - https://lucene.apache.org/)
     (Apache 2) Apache Lucene (module: spatial3d) (org.apache.lucene:lucene-spatial3d:9.4.2 - https://lucene.apache.org/)
     (The Apache License, Version 2.0) org.apiguardian:apiguardian-api (org.apiguardian:apiguardian-api:1.1.0 - https://github.com/apiguardian-team/apiguardian)
     (Apache 2.0) Awaitility (org.awaitility:awaitility:3.0.0 - http://awaitility.org)
     (The MIT License) Checker Qual (org.checkerframework:checker-qual:3.22.0 - https://checkerframework.org)
     (New BSD License) Hamcrest Core (org.hamcrest:hamcrest-core:1.3 - https://github.com/hamcrest/JavaHamcrest/hamcrest-core)
     (New BSD License) Hamcrest library (org.hamcrest:hamcrest-library:1.3 - https://github.com/hamcrest/JavaHamcrest/hamcrest-library)
     (The Apache Software License, Version 2.0) JetBrains Java Annotations (org.jetbrains:annotations:17.0.0 - https://github.com/JetBrains/java-annotations)
     (Eclipse Public License v2.0) JUnit Jupiter API (org.junit.jupiter:junit-jupiter-api:5.7.1 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Jupiter Engine (org.junit.jupiter:junit-jupiter-engine:5.7.1 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Jupiter Params (org.junit.jupiter:junit-jupiter-params:5.7.1 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Platform Commons (org.junit.platform:junit-platform-commons:1.7.1 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Platform Engine API (org.junit.platform:junit-platform-engine:1.7.1 - https://junit.org/junit5/)
     (Eclipse Distribution License - v 1.0) (Eclipse Public License, Version 2.0) org.locationtech.jts:jts-core (org.locationtech.jts:jts-core:1.19.0 - https://www.locationtech.org/projects/technology.jts/jts-modules/jts-core)
     (Apache License, Version 2.0) Proj4J (org.locationtech.proj4j:proj4j:1.1.5 - https://github.com/locationtech/proj4j)
     (The Apache Software License, Version 2.0) Spatial4J (org.locationtech.spatial4j:spatial4j:0.8 - https://projects.eclipse.org/projects/locationtech.spatial4j)
     (Apache 2) Objenesis (org.objenesis:objenesis:2.5.1 - http://objenesis.org)
     (The Apache License, Version 2.0) org.opentest4j:opentest4j (org.opentest4j:opentest4j:1.2.0 - https://github.com/ota4j-team/opentest4j)
     (BSD-2-Clause) PostgreSQL JDBC Driver (org.postgresql:postgresql:42.5.0 - https://jdbc.postgresql.org)
     (MIT) Duct Tape (org.rnorth.duct-tape:duct-tape:1.0.8 - https://github.com/rnorth/duct-tape)
     (Apache 2) org.roaringbitmap:RoaringBitmap (org.roaringbitmap:RoaringBitmap:0.9.38 - https://github.com/RoaringBitmap/RoaringBitmap)
     (Apache 2) org.roaringbitmap:shims (org.roaringbitmap:shims:0.9.38 - https://github.com/RoaringBitmap/RoaringBitmap)
     (MIT License) JUL to SLF4J bridge (org.slf4j:jul-to-slf4j:2.0.3 - http://www.slf4j.org)
     (MIT License) SLF4J API Module (org.slf4j:slf4j-api:2.0.3 - http://www.slf4j.org)
     (MIT) Testcontainers :: Database-Commons (org.testcontainers:database-commons:1.17.3 - https://testcontainers.org)
     (MIT) Testcontainers :: JDBC (org.testcontainers:jdbc:1.17.3 - https://testcontainers.org)
     (MIT) Testcontainers :: JUnit Jupiter Extension (org.testcontainers:junit-jupiter:1.17.3 - https://testcontainers.org)
     (MIT) Testcontainers :: JDBC :: PostgreSQL (org.testcontainers:postgresql:1.17.3 - https://testcontainers.org)
     (MIT) Testcontainers Core (org.testcontainers:testcontainers:1.17.3 - https://testcontainers.org)
     (ISC License) flatgeobuf (org.wololo:flatgeobuf:3.24.0 - https://github.com/flatgeobuf/flatgeobuf)
     (The Apache Software License, Version 2.0) SQLite JDBC (org.xerial:sqlite-jdbc:3.39.3.0 - https://github.com/xerial/sqlite-jdbc)
     (The Apache Software License, Version 2.0) nestedvm (org.xerial.thirdparty:nestedvm:1.0 - http://www.xerial.org/maven/repository/site/xerial/nestedvm)
     (Apache License, Version 2.0) SnakeYAML (org.yaml:snakeyaml:1.28 - http://www.snakeyaml.org)

[joshfischer1108]

Sorry, I've been out of the loop for awhile. I'm trying to catch up, I"ll probably ask a few not so smart questions along the way. 😄 .

I'll have to check my notes (if I can still find them) on why we did this with Heron. But we had a list of all licenses for dependencies that were not ALV2.

Some examples are:

• https://github.com/apache/airflow/tree/main/licenses
• https://github.com/apache/kafka/tree/trunk/licenses

[bchapuis]

As a total newbie to licencing myself, your questions really help me make sanity checks 😂

In my current understanding of the process, we must list the licenses of third-parties that have been integrated in the source code (this has been done in the LICENSE and DISCLAIMER files). I'm not sure if the Licenses of the dependencies must be listed.

[joshfischer1108]

Ahh. I see now. I was looking at the main branch of the repo and not the 492-release branch. Total. Facepalm. Moment. I guess we can say this falls in the category of "not so smart questions" 😂

[bchapuis]

This is also the reason why I'd like to merge this branch. In terms of collaboration and work distribution, it is probably preferable if the remaining issues are addressed separately in small pull requests.

[joshfischer1108]

I like small commits, but I'll leave that up to you and the others to decide.. I have a lot of catching up to do.

[bchapuis]

I also think small commits are the right approach. Do not hesitate to share your thoughts on how we could improve our organisation as an open source project. Few of us have experience in this area and it is critical for us to improve if we want to attract more contributors.

I will merge 492-release. I guess we should keep this issue open to track the overall advancement and convert the bullet points into separate issues labelled with "help needed" and "good first issue" when appropriate.

[swerky]

Hi,
I created a few issues from the discussion on the mailing list and the remarks from @julianhyde.
I'm not allowed to edit this issue to add them in the Misc category.

Here are the list of the issues:

• Check the licenses with the maven rat plugin #603
• Missing files in src release #604
• Add prefix apache- on the tar file names and on the directory inside the tar #605
• Add version number in README in the release #606
• Verify the projects necessary in the NOTICE file #607
• Verify and correct the LICENSE file for the -bin.tar release #608

[bchapuis]

Thanks, I added them to the list.