Remove excessive coupling between sonatype commands and publishSigned

[jvican]

In my experience, there's an indirect dependency between publishSigned and all the sonatype commands, meaning that if publishSigned is executed concurrently the whole sonatype release will fail, detecting changes coming from other repositories.

I don't know why this is done this way, and I couldn't find the exact way it's done (I think you're polling the repo info from xml files created by publishSigned?), but I think this is undesirable:

1. It forces a logical dependency between the two, when they can be decoupled.
2. It prevents, by nature, concurrent execution of tasks.

Is there any way we can remove such decoupling? I find the way sbt-bintray handles releases ideal. bintrayRelease is completely independent of publishSigned, and can be executed concurrently.

Maybe the solution to all this is to get rid of the commands and just have tasks, as bintray does...

[xerial]

@jvican I don't think there is any dependency between publishSigned and sonatypeRelease command. publishSigned is an sbt task, and sonatypeRelease is an independent command.

I don't know why you are seeing this dependency, but my expectation is running publishSigned concurrently for multiple modules, but I usually would like to run sonatypeRelease command only once.

If there is a proper way to define a task which can be defined only at the root module, we can make sonatypeRelease as a task, but I struggled to define a task that can be enabled only at the root module other than setting redundant task settings in every projects (e.g., ++ sonatypeSettings in root and ++ sonatypeDisableSettings in other sub modules, etc.)

[jvican]

The dependency comes from the fact that:

1. You cannot publishSigned concurrently. If a sonatypeRelease command is executed concurrently, it will fail to identify which module should be published.
2. The current workaround is to execute the following commands sequentially from a task ;sonatypeOpen bla;publishSigned;sonatypeRelease, and to make sure that they are exclusively for one project (two projects cannot release concurrently).
3. Even running ;publishSigned;sonatypeRelease fails. https://platform-ci.scala-lang.org/scalacenter/sbt-release-early/143.
4. sonatypeReleaseAll doesn't work and assumes that all the existing modules are aggregated in the root. Also, there's no way you can automate its execution. For example, sbt-release-early does not execute the releaseEarly task for the root project if it's not publishable.
5. I smell the root of the problem comes from all these things being a command and sharing the state. Wouldn't it be possible to distribute this state across keys scoped in the appropriate projects?

I don't know why you are seeing this dependency, but my expectation is running publishSigned concurrently for multiple modules, but I usually would like to run sonatypeRelease command only once.

This means there's no support for executing sonatypeRelease for a concrete module, because the id used to identify the publishing comes from Sonatype and not sbt. It also means that this is very unfriendly for sbt plugins automating publishing because it requires dependency analysis between the projects to find out who should run sonatypeReleaseAll, no?

In any case, having all these commands as tasks and decoupling the consumption of state would allow for the parallel execution of publish and sonatypeRelease, which are very long processes. On top of this, once you have them as tasks, you can always provide a command.

I'm not proposing a concrete fix to this situation yet, though I do give some hints. The goal of this comment is to make sure that we're on the same page and remove sources of miscommunication.

[xerial]

I think I understand your points, and a reason we cannot use tasks right now is sbt auto-plugin will enable sbtSonatypeRelease task for all projects. The early implementation of sbt-sonatype actually used tasks for sonatypeXXX, but gave it up because we cannot distinguish project types (root? aggregated? or sub-module?)

• If we have multiple modules that need to be published separately, we may need to explicitly enable plugin or add a setting to invoke sonatypeRelease task in each module root (an aggregated project).

  • Or if we may be able to skip running sonatypeRelease task in sub modules, for example, if 'sonatypePublish := true` is not set, but we usually want to enable this only when releasing the project so using settingKey might be inappropriate...

• For 5, I have no idea what scope will be appropriate. Sonatype staging profile ids will be issued after running publishSigned (or when creating a new session with sonatypeOpen). We need to find a way to pass (or link) the generated profile id between publishSigned and sonatypeRelease.

[maqdev]

I'm not sure if this is related but i've also found that executing sbt ";sonatypeOpen ABC; publishSigned" creates staging repository with description ABC, but then publishSigned ignores that repository and creates a new one with description "Implicitly created (autostaging)."

[maqdev]

I really don't understand how publishSigned choses the staging repository. Sometimes multiple parallel publishSigned commands choses one single (auto staging repository). But often they just creates new one...

[jvican]

I'm not sure if this is related but i've also found that executing sbt ";sonatypeOpen ABC; publishSigned" creates staging repository with description ABC, but then publishSigned ignores that repository and creates a new one with description "Implicitly created (autostaging)."

I've experienced the same, and I documented it here: jvican/sbt-release-early#18.

[lihaoyi]

@maqdev @jvican Mill's Sonatype publishing support always puts things in a single staging repository, and it will always only close/release the staging repository it published. Mill thus doesn't have any of the weird concurrency issues I used to keep hitting with SBT.

Mill's upload-to-sonatype and close-release-from-sonatype functions are just functions, and we pass the profile ID between them as a function parameter

[xerial]

sbt-sonatype 3.0 supports concurrent publishSigned by using sonatypeSessionName as a unique identifier: https://github.com/xerial/sbt-sonatype#uploading-artifacts-in-parallel

@jvican
I'm not sure this can fully support your specific use cases, but I'm now actively testing this use case in my own project wvlet/airframe#663

Let me close this ticket now that we have a solution for this.