[gradle] Reworking publishing pipeline #2886

jimschubert · 2019-05-14T03:36:09Z

PR checklist

Read the contribution guidelines.
Ran the shell script under ./bin/ to update Petstore sample so that CIs can verify the change. (For instance, only need to run ./bin/{LANG}-petstore.sh, ./bin/openapi3/{LANG}-petstore.sh if updating the {LANG} (e.g. php, ruby, python, etc) code generator or {LANG} client's mustache templates). Windows batch files can be found in .\bin\windows\. If contributing template-only or documentation-only changes which will change sample output, be sure to build the project first.
Filed the PR against the correct branch: master~~, 3.4.x, 4.0.x~~. Default: master.
Copied the technical committee to review the pull request if your PR is targeting a particular programming language.

Description of the PR

TravisCI proxies separate external requests with different orginating IP
addresses, while Sonatype associates artifacts for auto-generated
repositories by IP address. This leads to many gradle deploys from CI
resulting in "split" staging repositories with no way to combine in
Sonatype Nexus.

This introduces a workflow which should resolve this issue on the next
revision release. Specifically, nexus-publish-plugin is included to
create singular staging repositories from TravisCI and
gradle-nexus-staging-plugin is included to auto-release and promote
this repository.

NOTE:
We need to publish via publishPluginMavenPublicationToNexusRepository,
because publishToNexus will publish all publish-related tasks,
including the one intended only for the Gradle Plugin Portal.

Tested in standalone open source Nexus Repo Manager, which doesn't
support the staging plugin. So, only SNAPSHOT workflow has been
validated locally.

The enableFeaturePreview('STABLE_PUBLISHING') option in settings.gradle is a warning presented by the new nexus-publish-plugin:

As part of making the publishing plugins stable, the 'deferred configurable' behavior of the 'publishing {}' block has been deprecated. In Gradle 5.0 the 'enableFeaturePreview('STABLE_PUBLISHING')' flag will be removed and the new behavior will become the default. Please add 'enableFeaturePreview('STABLE_PUBLISHING')' to your settings file and do a test run by publishing to a local repository. If all artifacts are published as expected, there is nothing else to do. If the published artifacts change unexpectedly, please see the migration guide for more details: https://docs.gradle.org/4.10.2/userguide/publishing_maven.html#publishing_maven:deferred_configuration.

This PR is an attempt at resolving #411
cc @OpenAPITools/generator-core-team

TravisCI proxies separate external requests with different orginating IP addresses, while Sonatype associates artifacts for auto-generated repositories by IP address. This leads to many gradle deploys from CI resulting in "split" staging repositories with no way to combine in Sonatype Nexus. This introduces a workflow which should resolve this issue on the next revision release. Specifically, nexus-publish-plugin is included to create singular staging repositories from TravisCI and gradle-nexus-staging-plugin is included to auto-release and promote this repository. NOTE: We need to publish via publishPluginMavenPublicationToNexusRepository, because publishToNexus will publish _all_ publish-related tasks, including the one intended only for the Gradle Plugin Portal. Tested in standalone open source Nexus Repo Manager, which doesn't support the staging plugin. So, only SNAPSHOT workflow has been validated locally.

auto-labeler · 2019-05-14T03:36:14Z

👍 Thanks for opening this issue!
🏷 I have applied any labels matching special text in your issue.

The team will review the labels and make any necessary changes.

jimschubert · 2019-05-14T03:36:55Z

.travis.yml

@@ -136,8 +136,8 @@ after_success:
        echo "Finished mvn clean deploy for $TRAVIS_BRANCH";
        pushd .;
        cd modules/openapi-generator-gradle-plugin;
-        ./gradlew -Psigning.keyId="$SIGNING_KEY" -Psigning.password="$SIGNING_PASSPHRASE" -Psigning.secretKeyRingFile="${TRAVIS_BUILD_DIR}/sec.gpg" -PossrhUsername="${SONATYPE_USERNAME}" -PossrhPassword="${SONATYPE_PASSWORD}" uploadArchives --no-daemon;
-        echo "Finished ./gradlew uploadArchives";


If the 4.0.1 release fails, we may consider reverting to the uploadArchives and maven publishing plugin.

jmini · 2019-05-15T05:53:57Z

.travis.yml

-        ./gradlew -Psigning.keyId="$SIGNING_KEY" -Psigning.password="$SIGNING_PASSPHRASE" -Psigning.secretKeyRingFile="${TRAVIS_BUILD_DIR}/sec.gpg" -PossrhUsername="${SONATYPE_USERNAME}" -PossrhPassword="${SONATYPE_PASSWORD}" uploadArchives --no-daemon;
-        echo "Finished ./gradlew uploadArchives";
+        ./gradlew -Psigning.keyId="$SIGNING_KEY" -Psigning.password="$SIGNING_PASSPHRASE" -Psigning.secretKeyRingFile="${TRAVIS_BUILD_DIR}/sec.gpg" -PossrhUsername="${SONATYPE_USERNAME}" -PossrhPassword="${SONATYPE_PASSWORD}" publishPluginMavenPublicationToNexusRepository --no-daemon;
+        echo "Finished ./gradlew publishPluginMavenPublicationToNexusRepository";


We need to see how this goes, but I guess that calling the task closeAndReleaseRepository would also be necessary. (the maven nexus plugin is doing something like this)

This is actually the SNAPSHOT path (master), but your comment made me realize that I had completely missed updating the uploadArchives in the following conditional block which is used for releases. I believe the releases branch is the only one that requires closing.

jmini · 2019-05-15T05:59:55Z

Because of the change maven to maven-publish I have updated the helper-pom to use the task publishToMavenLocal, see 6b0c3fa

jmini · 2019-05-15T06:02:27Z

Now it is failing because of this error:

Cannot perform signing task ':signMavenJavaPublication' because it has no configured signatory

I think there is a way to say that signing is only for release version:
https://docs.gradle.org/current/userguide/signing_plugin.html#sec:conditional_signing

jimschubert · 2019-05-15T13:36:34Z

@jmini thanks for also looking into this. Seems I'll need to work out the release signing for maven-publish. I'll try and get to that on the next day or so.

jimschubert · 2019-05-15T18:12:52Z

In digging a little more, it seems we will need to call closeRepository on master (for snapshots) and closeAndReleaseRepository on versioned branches (for releases).

Here's the task graph of the target task:

\--- :publishPluginMavenPublicationToNexusRepository
     +--- :generatePomFileForPluginMavenPublication
     +--- :initializeNexusStagingRepository
     +--- :jar
     |    +--- :classes
     |    |    +--- :compileJava
     |    |    |    \--- :compileKotlin
     |    |    \--- :processResources
     |    |         \--- :pluginDescriptors
     |    +--- :compileKotlin
     |    \--- :inspectClassesForKotlinIC
     |         +--- :classes
     |         |    +--- :compileJava
     |         |    |    \--- :compileKotlin
     |         |    \--- :processResources
     |         |         \--- :pluginDescriptors
     |         \--- :compileKotlin
     \--- :signPluginMavenPublication
          +--- :generatePomFileForPluginMavenPublication
          \--- :jar
               +--- :classes
               |    +--- :compileJava
               |    |    \--- :compileKotlin
               |    \--- :processResources
               |         \--- :pluginDescriptors
               +--- :compileKotlin
               \--- :inspectClassesForKotlinIC
                    +--- :classes
                    |    +--- :compileJava
                    |    |    \--- :compileKotlin
                    |    \--- :processResources
                    |         \--- :pluginDescriptors
                    \--- :compileKotlin

initializeNexusStagingRepository will create the repository with our defined staging id. iirc from discussion with an engineer at Sonatype, these repositories are cleaned after 24 hours anyway, but I wouldn't want an open "snapshot" repository to become mixed in anyway with a release repository on the same day.

…task guard, and publish to Gradle Plugin portal only on tagged build.

* 4.1.x: (56 commits) sync master Update compatibility table Ruby client: escape path parameters (OpenAPITools#3039) [gradle plugin] Release 4.0.1 fixes (OpenAPITools#3051) Update version to 4.0.2-SNAPSHOT (OpenAPITools#3047) Map number to double time since float is also parsed as double in Qt5 C++ (OpenAPITools#3046) Prepare 4.0.1 release (OpenAPITools#3041) [gradle] Reworking publishing pipeline (OpenAPITools#2886) [typescript-fetch] Fix uploading files (OpenAPITools#2900) Resolves OpenAPITools#2962 - Add properties config to Maven parameters (OpenAPITools#2963) fix(golang): Check error of xml Encode (OpenAPITools#3027) [C++][Restbed] Add handler callback methods (OpenAPITools#2911) Remove null checks for C# value types (OpenAPITools#2933) [python-server] Support python 3.7 for all server-generators (OpenAPITools#2884) Use golang's provided method names (gin) (OpenAPITools#2983) [python] Adding constructor parameters to Configuration and improving documentation (OpenAPITools#3002) Add new option to maven plugin's readme (OpenAPITools#3025) Engine param in maven plugin. (OpenAPITools#2881) Added support for patterns on model properties (OpenAPITools#2948) [csharp] Make API response headers dictionary case insensitive (OpenAPITools#2998) ...

jimschubert added the Enhancement: CI/Test label May 14, 2019

auto-labeler bot added the OpenAPI Generator Gradle Plugin label May 14, 2019

jimschubert commented May 14, 2019

View reviewed changes

Change the 'publish' task to 'publishToMavenLocal' in the helper pom.xml

6b0c3fa

jmini reviewed May 15, 2019

View reviewed changes

jimschubert added 2 commits May 15, 2019 10:30

Add condition on signing and be explicit about sources/javadoc artifacts

becc3b2

Call close/closeAndReleaseRepository

a8d4ec0

jimschubert added 2 commits May 25, 2019 07:24

Reuse same closeAndReleaseRepository for SNAPSHOT + releases using a …

59c0ea1

…task guard, and publish to Gradle Plugin portal only on tagged build.

Add v prefix for tag check.

9049b2a

wing328 added this to the 4.0.1 milestone May 31, 2019

wing328 merged commit 6a1fc51 into master May 31, 2019

wing328 deleted the gradle-plugin-publishing branch May 31, 2019 12:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[gradle] Reworking publishing pipeline #2886

[gradle] Reworking publishing pipeline #2886

jimschubert commented May 14, 2019 •

edited

Loading

auto-labeler bot commented May 14, 2019

jimschubert May 14, 2019

jmini May 15, 2019

jimschubert May 15, 2019

jmini commented May 15, 2019

jmini commented May 15, 2019

jimschubert commented May 15, 2019

jimschubert commented May 15, 2019

[gradle] Reworking publishing pipeline #2886

[gradle] Reworking publishing pipeline #2886

Conversation

jimschubert commented May 14, 2019 • edited Loading

PR checklist

Description of the PR

auto-labeler bot commented May 14, 2019

jimschubert May 14, 2019

Choose a reason for hiding this comment

jmini May 15, 2019

Choose a reason for hiding this comment

jimschubert May 15, 2019

Choose a reason for hiding this comment

jmini commented May 15, 2019

jmini commented May 15, 2019

jimschubert commented May 15, 2019

jimschubert commented May 15, 2019

jimschubert commented May 14, 2019 •

edited

Loading