Create a sbt-paradox-pekko sbt plugin

[mdedetrich]

With #91 we have managed to replace the sbt-paradox-akka theme with the standard one from paradox however it was just done by manually putting in all of the sbt-paradox-* plugins. Ideally however we would like to create our own sbt-paradox-pekko sbt plugin akin to sbt-paradox-akka. This will become especially important when we actually want to create our own bespoke pekko doc theme which we will share within all of our pekko modules.

The only slightly complicating factor is that currently pekko core sbt build enforces JDK 1.8 (sbt doesn't even load unless SBT is its run with JDK 1.8 on the path). The problem here is due to the design of one of sbt-paradox's core dependencies parboiled, you can only use specific versions of parboiled on JDK 1.8 versus later. In summary, pekko core only only use an outdated set of sbt-paradox-* dependencies where as other pekko modules which are built with newer JDK's will only support the newer sbt-paradox-* dependencies.

Thankfully I have managed to resolve this issue, assuming you have sbt-paradox-pekko with the newer sbt-paradox-* dependencies you can do this in pekko core's build.sbt

// We have to deliberately use older versions of sbt-paradox because current Pekko sbt build
// only loads on JDK 1.8 so we need to bring in older versions of parboiled which support JDK 1.8
addSbtPlugin("org.apache.pekko" % "sbt-paradox-pekko" % "HEAD+20230107-1453" excludeAll(
  "com.lightbend.paradox","sbt-paradox",
  "com.lightbend.paradox" % "sbt-paradox-apidoc",
  "com.lightbend.paradox" % "sbt-paradox-project-info"
))
addSbtPlugin("com.lightbend.paradox" % "sbt-paradox" % "0.9.2" force())
addSbtPlugin("com.lightbend.paradox" % "sbt-paradox-apidoc" % "0.10.1" force())
addSbtPlugin("com.lightbend.paradox" % "sbt-paradox-project-info" % "2.0.0" force())

I already have a prototype sbt-paradox-pekko plugin locally, it doesn't implement any theme but at least it brings in the sbt-paradox-* dependencies and will allow us to enforce a consistent theme.

• Upload sbt-paradox-pekko sbt plugin from local git repo to a temporary org
• Create an INFRA ticket to setup an incubator-sbt-paradox-pekko git repo
• Publish the sbt-paradox-pekko sbt plugin to the apache maven repo so it can be used in pekko repos. Can also use this simple project as a test bed for Automate the release #85

[pjfanning]

you don't need an INFRA ticket - they have created https://selfserve.apache.org/ - I can create the extra repo.

[pjfanning]

created https://github.com/apache/incubator-pekko-sbt-paradox - it has to start with incubator-pekko - the main branch is not yet protected

[mdedetrich]

Thanks for organizing this! Will get onto it in the next few days

[jrudolph]

The problem here is due to the design of one of sbt-paradox's core dependencies parboiled, you can only use specific versions of parboiled on JDK 1.8 versus later.

Ok, that's because the latest paradox depends on parboiled 1.4.1 which is the first version to remove Java 8 support and consequently paradox itself had to remove Java 8 support (otherwise we could just pin parboiled to 1.4.0 here). I guess that was something of an oversight when releasing paradox 0.10.0 because it could (and maybe should) have stayed on parboiled 1.4.0 to keep Java 8 compatibility (if only to make upgrading to latest paradox easier for Akka itself going forward).

[pjfanning]

The incubator-pekko-sbt-paradox plugin will need to add the following to the HTML pages we generate.

• An Apache copyright statement (probably at bottom of each page) - eg https://spark.apache.org/
• A tab or menu (probably accessible on all pages) that provides links for pre-existing Apache pages
  • Apache home page
  • Sponsoring
  • License
  • Trademark
  • Security
  • Thanks
  • Upcoming Apache Event (a gif - or a link to a page)
  • see the 'Apache Software Foundation' menu at top right of https://spark.apache.org/
• Other tabs/menus for (see top banner on https://spark.apache.org/)
  • Downloads
  • Developer Resources
  • Community
  • we will need to create the pages for these items to link to
• An Apache Pekko logo (probably including 'Apache Pekko' name) - see top left of https://spark.apache.org/
• A favicon
• The Akka docs have a search box, while https://pekko.apache.org does not
• ideally, we would add a comment to all the generated HTML, Javascript, etc. which contains an Apache license header - if this is hard, we might be able to defer this

[jrudolph]

I'm looking into setting this up. One problem, is that we have a kind of chicken and egg problem here, that AFAIU we cannot easily just make a public release of any kind of plugin even if it's just for internal consumption (especially, as it will likely need some iterations to stabilize). What options do we have?

• publish snapshots of the plugin to nightly repo and use those
• include sbt plugin by source reference (will slow down starting sbt quite a bit, as it will then include the full plugin sbt as well)
• include the plugin sources as a git submodule into the sbt builds (faster than source reference but will be the first submodule, so extra steps needed while cloning)
• defer using a plugin for the time being until we have released at least a first version of the main pekko repo and just put everything in the build here (some extra work to make it work in other modules later either by manual copying or by figuring out a better setup later on)

I'd start with setting docs up in the main repo first, so that we are not blocked by a missing docs page for the first release.

[mdedetrich]

I'm looking into setting this up.

So I already have a local plugin as a template, should I just send it over to you or should I just push what I have? What I have locally doesn't have any theme, it just contains the other sbt plugins (i.e. project-info) and its been adapted from https://github.com/akka/akka-paradox

One problem, is that we have a kind of chicken and egg problem here, that AFAIU we cannot easily just make a public release of any kind of plugin even if it's just for internal consumption (especially, as it will likely need some iterations to stabilize). What options do we have?

To me the most pragmatic solution would be to just publish it as a snapshot as the org.apache.pekko org into the Apache Nexus Maven repo just as we have done with Pekko. I would set it up to just publish snapshots after something is pushed into main, that way its not so annoying to deal with incremental snapshot versions.

[justinmclean]

Hi, Out of interest what’s stopping you from making a pubic source release? Justin

[jrudolph]

Out of interest what’s stopping you from making a pubic source release?

We need a ivy/Maven binary release of the plugin to be able to consume it (which is somewhat difficult in itself, because sbt plugins usually need an ivy repository to publish to (though, Maven usually also works even if it uses non-standard naming)).

In general, I do not exactly know what it would entail to make a public release of any kind for a tool used for only developer consumption (does it have to go through the IPMC voting process etc.?) but I also do not have time to figure that out right for something that is just a potential convenience later on.

[mdedetrich]

Out of interest what’s stopping you from making a pubic source release?

The reason why we are not looking at a public source release initially, is that similar to a JVM library like Apache Commons dependency resolution rely on resolving artifacts from Maven, not from a local source directory. Specifically with sbt, it uses addSbtPlugin which works by resolving a maven repository (see https://www.scala-sbt.org/1.x/docs/Using-Plugins.html). Do note that this is an plugin for sbt which is a build tool, its not going to be code that will be executed as part of using the library.

I would imagine this is not dissimilar to projects like Apache Commons which even though has to make a source release to comply with ASF guidelines, since its a library the vast majority of users are not going to download the source package. They will add it as a dependency in their pom.xml, build.sbt, build.grade etc etc which will download the binary from ASF Nexus repo.

This is one of the disconnects I was talking about earlier, if the Apache Project happens to be a library and not an "executable application" almost no users will download the source package and in cases like this its not even technically feasible (or desirable) to use a source package and Pekko is not unique in this regard, other TLP's which are libraries like Apache Commons are also treated in the same way.

[justinmclean]

Hi, Apache makes source releases, binaries can be created to help users but they are optional. 3rd parties (which could involve people in the project) can also provide binary releases based on Apache source releases. This may help https://infra.apache.org/publishing-maven-artifacts.html <https://infra.apache.org/publishing-maven-artifacts.html> Kind Regards, Justin

[mdedetrich]

A source release package can be made for this sbt plugin, but doing it right now is incredibly premature. Its also an exceptional case because we are dealing with a plugin whose goal is to remove boilerplate from pekko sbt projects. If we just repeated the boilerplate that this plugin would be abstracting over in every sbt build definition we wouldn't even be having this discussion.

Apache makes source releases, binaries can be created to help users but they are optional. 3rd parties (which could involve people in the project) can also provide binary releases based on Apache source releases.

In the case of JVM libraries, from the perspective of users the binaries are not optional because the entire JVM ecosystem is built around resolving libraries as binaries. Almost all users in this context will ignore source packages, not willingly or out of malice but because none of the build tools for JVM projects are built around and/or expect source packages. This especially so in the case of sbt, as I said before sbt plugins only work with binaries published to maven.

This is the disconnect I am talking about and in my view it would be more appropriate to discuss this in a proper channel (I don't even know what channel this is, INFRA?)

[jrudolph]

Let's not get misled that we need a release (in Apache terms) of the plugin discussed here. The whole idea of the plugin is just an optimization for us developers. In the end, creating a package of the plugin is just like caching some intermediate results, so let's treat it like that and try to find a solution that does not require an Apache release or which would treat the plugin as a full-fledged module of pekko.

[mdedetrich]

Let's not get misled that we need a release (in Apache terms) of the plugin discussed here. The whole idea of the plugin is just an optimization for us developers. In the end, creating a package of the plugin is just like caching some intermediate results, so let's treat it like that and try to find a solution that does not require an Apache release or which would treat the plugin as a full-fledged module of pekko.

Agreed, I also do not want to treat this as an official ASF project. The best way to think of it is a common script that is shared between multiple Pekko projects.

There might (and real emphasis on might) be a reason to make it a proper release, but this would be way down the line, again re-iterating its premature now.

[pjfanning]

One option would be for someone to just create this repo as a personal, non-ASF, repo.

I don't see the harm in keeping this as an ASF repo and using snapshot jars for the next while, while the plugin code is still changing a lot. We could later do a 1.0.0 ASF release and include bin and src distributions. Once we learn how to build those for the main pekko repos, it should be straightforward to apply that here.

FYI, I plan to develop https://github.com/pjfanning/sbt-source-dist as a non-ASF project (at least, for now).

[mdedetrich]

FYI, I plan to develop https://github.com/pjfanning/sbt-source-dist as a non-ASF project (at least, for now).

I was also planning to develop/prototype a sbt-apache-release plugin which would use sbt-sonatype/sbt-pgp along with your new sbt-source-dist as a way to collect all of the steps (that make sense to do so) to create a single sbt release step. I will wait until your sbt-source-dist plugin is usable in sbt though

[mdedetrich]

@jrudolph I have just pushed a template to https://github.com/apache/incubator-pekko-sbt-paradox and also created an INFRA ticket to set it up for Apache Nexus deployment (see https://issues.apache.org/jira/browse/INFRA-24157)

[pjfanning]

https://github.com/apache/incubator-pekko-sbt-paradox exists - we should use that repo (issues/PRs/discussions) to track status of that work

[mdedetrich]

@justinmclean I just had a discussion with @Claudenw so I just want to clarify some things.

Earlier when I (and others) was talking about a ivy/Maven binary release, this terminology might be a bit confusing because it doesn't just contain a binary (specifically for JVM we are talking about .class files) but also sources and other files. To illustrate this, in the https://github.com/apache/incubator-pekko-sbt-paradox project if you run sbt publishLocal it will create the artifacts and deploy them into a local repository, i.e.

[info] :: delivering :: org.apache.pekko#pekko-sbt-paradox;0.0.0+18-8f2ba638-SNAPSHOT :: 0.0.0+18-8f2ba638-SNAPSHOT :: integration :: Wed Feb 15 11:39:15 CET 2023
[info] 	delivering ivy file to /Users/mdedetrich/github/incubator-pekko-sbt-paradox/plugin/target/scala-2.12/sbt-1.0/ivy-0.0.0+18-8f2ba638-SNAPSHOT.xml
[info] 	published pekko-sbt-paradox to /Users/mdedetrich/.ivy2/local/org.apache.pekko/pekko-sbt-paradox/scala_2.12/sbt_1.0/0.0.0+18-8f2ba638-SNAPSHOT/poms/pekko-sbt-paradox.pom
[info] 	published pekko-sbt-paradox to /Users/mdedetrich/.ivy2/local/org.apache.pekko/pekko-sbt-paradox/scala_2.12/sbt_1.0/0.0.0+18-8f2ba638-SNAPSHOT/jars/pekko-sbt-paradox.jar
[info] 	published pekko-sbt-paradox to /Users/mdedetrich/.ivy2/local/org.apache.pekko/pekko-sbt-paradox/scala_2.12/sbt_1.0/0.0.0+18-8f2ba638-SNAPSHOT/srcs/pekko-sbt-paradox-sources.jar
[info] 	published pekko-sbt-paradox to /Users/mdedetrich/.ivy2/local/org.apache.pekko/pekko-sbt-paradox/scala_2.12/sbt_1.0/0.0.0+18-8f2ba638-SNAPSHOT/docs/pekko-sbt-paradox-javadoc.jar

As you can see it creates a pekko-sbt-paradox-sources.jar and this jar file does contain the sources used to build the plugin (actually in contrast to a standard Apache source distribution, the *-sources.jar file is far more accurate because it contains ONLY the sources which are used to build the plugin and nothing else). For each of these jar files there is also .sha1 and .md5 hashes, i.e.

<~/.i/l/o/p/s/s/0/srcs>-> ls
pekko-sbt-paradox-sources.jar       pekko-sbt-paradox-sources.jar.md5   pekko-sbt-paradox-sources.jar.sha1

Additionally if you were to create a signed package (i.e. publishLocalSigned) which signs it via gpg/pgp it would also contain expected signature files (such as .asc) which you can use to verify the signature.

I have attached an example distribution 0.0.0+18-8f2ba638-SNAPSHOT.zip, note that .jar files are just standard zip files if you want to introspect them.

In summary in regards to sbt and hence pekko and these plugins that we talk about, there is nothing special/bespoke/unique about this. The sbt build tool by default will make sure that not only the binaries but the sources and other relevant meta information are distributed in Maven Repositories (whether that be Apache's Maven repo, Sonatypes OSS repo or a local folder repository as used in the previous example).

The only exception to this is the various LICENSE/NOTICE/DISCLAIMER and other files which as you have noticed from the other repositories are being handled additionally.

[pjfanning]

Since this is an Apache project, we will still need to produce source releases (using sbt-source-dist) and go through the release process, including votes in the Pekko PPMC and the Incubator PMC.

I don't see anything significantly different in the Maven publish of this project - compared to incubator-pekko, for instance.

[mdedetrich]

Since this is an Apache project, we will still need to produce source releases (using sbt-source-dist) and go through the release process, including votes in the Pekko PPMC and the Incubator PMC.

Yes, I was just clarifying that when we talked about a "Maven binary release", its strictly speaking not just a pure binary.

I don't see anything significantly different in the Maven publish of this project - compared to incubator-pekko, for instance.

Agreed