Investigate correlation between artifacts upload and staging repository ID

[szpak]

Gradle doesn't provide any information about an ID of a staging repository (in a case of Nexus) where artifacts have been uploaded. It is needed to (optionally) correlate uploaded artifacts with a staging repository which should be closed/promoted by gradle-nexus-staging-plugin. It in turns is needed to make it possible have more than one open (closed) staging repository in Nexus for the same staging profile (see here and there why it is needed).

[akomakom]

I'm investigating this as well, so here are my notes on how that could be accomplished properly (current implementation relies on the Staging Repository Profile name matching the packageGroup exactly, which is both limiting and unintuitive.)

1. Retrieve the repo target configurations (/service/local/repo_targets)
2. Find one with a regex matching the configured packageGroup.
3. Retrieve the staging profiles (already being done), find the one referencing this repo target.

From what I can tell, this is likely to be very slow (for our sonatype account, step 1 takes more than 30 seconds).

That said, I just realized that (as a hackaround) I can just set the value of the packageGroup parameter to match the name of the target repository... but that would be hardcoding.

[deanhiller]

has anyone figured this out. I keep getting http 500 response from nexus recently :( and the http client nor the plugin is logging the body output (in case it has more error info in it).

[deanhiller]

oh another thought. Doing org.gradle.parallel=true actually causes all my uploads to fail in that it creates multiple staging repos instead of 1 :( with the maven plugin not this plugin, but I thought you guys may have run into that as well. parallel=true could bring my upload time down from 10 minutes to like 2 minutes but it doesn't seem to work. do you guys know how to work around by setting my own sonus staging identifier or something?

[szpak]

and the http client nor the plugin is logging the body output

Enable debug and the whole communication should be logged (together with digest auth, so do not attach the whole debug output as it is in this issues :).

Parallel is tricky with artifacts upload/CD in general. Artifacts from some modules could be already uploaded while tests in other module failed a little bit later and in the end not all of the artifacts (e.g. SNAPSHOTs) have been uploaded.

@deanhiller I propose you to raise an issue on Gradle forum about multiple staging repositories created in Nexus with upload and parallel.

[szpak]

@akomakom Thanks for sharing your insights. One of my ideas is to get the artifacts contained in the repository and take into consideration only those with a matching artifacts name (prefix). That would allow to use the plugin with multiple projects released in CD model concurrently (however, I haven't verified yet if that information is available via web API, but rather should be).

[deanhiller]

@szpak Actually, I sent you a pull request as the current plugin doesn't log the error :( :(. I am using my own version now in my project and it works great. Please see that pull request.

On the other issue. I am wondering if this ticket AND my parallel issue are BOTH solved by doing the following(I have been playing with the nexus api with a REST client today)....

Note the id here 683f50db65952 is fixed and is my projects permanent id...(I say we just put that in settings to avoid the extra call to nexus).

POST https://oss.sonatype.org/service/local/staging/profiles/683f50db65952/start
along with xml

<?xml version="1.0" encoding="UTF-8"?>
<promoteRequest>
  <data>
    <stagedRepositoryId>1090</stagedRepositoryId>
    <description>Temporary Webpieces Staging Repo</description>
    <targetRepositoryId>releases</targetRepositoryId>
  </data>
</promoteRequest>

This resulted in this response giving me the staging id to upload all artifacts too (note that the requested one didn't take for some reason :( )

<promoteResponse>
<data>
<stagedRepositoryId>orgwebpieces-1086</stagedRepositoryId>
<description>Temporary Webpieces Staging Repo</description>
 </data>
</promoteResponse>

Please realize whether I upload sequentially OR parallel, I am left in a bad state when it fails 10 artifacts in out of 20......I don't really care as either way it's bad and generally doesn't happen thankfully.

Once we have the staging id, I now can upload all artifacts in parallel to that one staging id!!!! In fact, in parallel, if I don't have a staging id, it auto-creates 2 different ones and starts re-using either or and uploading to two locations(it's very annoying).

This would allow parallel to work again AND it would solve this ticket as well. I just got two birds stoned ;).....errr. I mean one stone two birds but I like getting the birds stoned instead ;).

[deanhiller]

oh, and if this does get implemented, I will use it here https://github.com/deanhiller/webpieces and you can just point there for the example as well ;). I basically turn off parallel for my release.sh script but otherwise I run gradle builds in parallel as it goes from 2min30sec to around 55seconds.

[deanhiller]

I filed this ticket as well on the request / response issue

https://issues.sonatype.org/browse/OSSRH-27170

[szpak]

Thanks for sharing your findings, I had to miss start operation. Assuming all 3 commands to be executed in one Gradle run the value of stagedRepositoryId returned from the server could be passed to the close & promote tasks (and then even OSSRH-27170 should not be a problem).

[szpak]

Btw, projects using start command in automatic releases would need to take care of closing/dropping that repository on a failure (e.g. with a finalizer task). As an alternative it would be needed to force Gradle upload task to put artifacts only into that one particular staging repository (or fail).

[szpak]

#76 triggers a reaction and it's supported with external plugin which enhances upload/publish tasks in Gradle - nexus-publish-plugin.

See that comment for details.