Skip to content

[JSDK-9] Public releases and CI changes#79

Merged
tl-andrea-dilisio merged 60 commits intomainfrom
jsdk-9
Mar 4, 2022
Merged

[JSDK-9] Public releases and CI changes#79
tl-andrea-dilisio merged 60 commits intomainfrom
jsdk-9

Conversation

@tl-andrea-dilisio
Copy link
Contributor

@tl-andrea-dilisio tl-andrea-dilisio commented Feb 25, 2022

Disclaimer

  • Please don't be scared by the number of changes: most of them are either file or packages renames

Contents

  • Gradle changes required to publish on Sonatype
  • GH Workflow changes to the release process, see below. moreover a few DRY and caching improvements
  • Removed the existing lib module, the library is now in the root of the project
  • Renamed package names to comply with Maven conventions and our groupId com.truelayer

Release process changes proposal

Current problems

  • as we don't have gitops here, we cannot trust manual tags and let them trigger public release pipelines. By releasing bad artifacts with final versions we would end up with immutable artifacts on Maven Central, that cannot be deleted once released 😱
  • also, nothing would really prevent the creation of a tag from non main branches
  • We could generate tags as part of a WF here, but we can't prevent someone from pushing tags right now. So by listening on tags as triggers for releases we might end up in trouble. Protected tags are in private beta apparently, this might change in future

Proposed changes

General changes

  • Stop creating releases on new tags push event (i.e. manual operations that no one can review)
  • Leverage the project version key contained in the gradle.properties file as version-to-be-deployed
  • Both snapshot and final releases will be made available upon successful unit, integration and acceptance tests.

Unstable releases

  • Unstable releases will be managed via our Sonatype snapshots repository, and they will be available without any credentials to every developer that declares the public Sonatype Snapshot repository (https://s01.oss.sonatype.org/content/repositories/snapshots/) amongst the repositories and a dependency to our lib following the SNAPSHOT format, i.e. com.truelayer:trulayer-java:0.0.2-SNAPSHOT. With this approach we can release to external devs new artifacts without the need of explicit releases/tags
  • Similarly to final releases, gradle.properties file will have to contain a version in the $major.$minor.$build format, ie without -SNAPSHOT suffixes. This is meant to ease the merge to the main branch later: we don't have to change the version right before or after merging to the main branch
  • The declared version should not be already released as final version on Maven Central. For instance: if we already released a com.truelayer:trulayer-java:0.0.3 a workflow with version com.truelayer:trulayer-java:0.0.3 in the gradle.properties will fail - the following HTTP request has to return 404: https://repo1.maven.org/maven2/com/truelayer/truelayer-java/$version/truelayer-java-$version.pom will see if it makes sense in future
  • A new snapshot (according to what declared in the gradle.properties file ) will be released on every push on non main branches. Our CI will add a -SNAPSHOT suffix to the declared version and display on the GH WF what snapshot version we're packaging and publishing exactly. Gradle will automatically replace the SNAPSHOT label with the current timestamp at artifact packaging time. The same tooling will make sure that the latest timestamp available will be used when declaring a SNAPSHOT dependency in a project.
  • To create a final release it will be enough to merge a PR in the main branch

Final releases

  • Final release will be published to the main Sonatype repo and at the same time on Maven Central, on every push on the main branch
  • The first step after the usual checkout and gradle wrapper validation will be to verify the project version against the Maven Central repository. If a final release already exists with the same version the WF will fail. the following HTTP request has to return 404: https://repo1.maven.org/maven2/com/truelayer/truelayer-java/$version/truelayer-java-$version.pom will see if it makes sense in future
  • Once build and tests are passed, the release phase will kick off.
  • we'll create a tag, with this action: https://github.com/marketplace/actions/github-tag. The tag will follow what declared on the gradle.properties file
  • Then we'll create a release here as well with an automatically generated changelog, using this action: https://github.com/softprops/action-gh-release - the purpose of this is to convey context about the release
  • We can manually amend the changelog later if required.
  • Once all the above are passed, the nexus publishing task will be triggered. the outcome of this last task will be an artifact released on Maven Central

Be careful

❗ Before making our repo public we should triple check that external collaborators can only contribute by creating a fork and a PR originating from that. Moreover, we should make sure that external collaborator can't access our repo secrets nor execute GH workflow without our explicit approval. See this chapter for more details.

Comment on lines +154 to +165
task sourcesJar(type: Jar) {
from sourceSets.main.allJava
archiveClassifier = 'sources'
}

task javadocJar(type: Jar) {
from javadoc
archiveClassifier = 'javadoc'
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

required to publish to maven central

Comment on lines +208 to +217
signing {
def signingKey = System.getenv('SONATYPE_GPG_KEY')
def signingPassword = System.getenv('SONATYPE_GPG_PASSPHRASE')
useInMemoryPgpKeys(signingKey, signingPassword)
sign publishing.publications.mavenJava
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

required by maven central

@tl-luca-baggi
Copy link
Contributor

Generally speaking LGTM, just a couple of notes reg unstable releases:

  • relying on pushing the branch to create a new SNAPSHOT release, there is the risk to release an incomplete or not working feature if the tests are still passing. So I think we should make crystal clear for external devs the fact that declaring a dependency from SNAPSHOT can cause their solution to break quite often and without notice
  • somehow related to the point above, if we work in parallel on different branches (and different features), there is the risk to push intermittently SNAPSHOT releases with same version but quite different among them

@tl-andrea-dilisio
Copy link
Contributor Author

tl-andrea-dilisio commented Mar 2, 2022

  • relying on pushing the branch to create a new SNAPSHOT release, there is the risk to release an incomplete or not working feature if the tests are still passing. So I think we should make crystal clear for external devs the fact that declaring a dependency from SNAPSHOT can cause their solution to break quite often and without notice

I think this is a valid point, but AFAIK this is the direction we want to take with unstable releases going forward: we don't want to keep them secret, as long as they're auto consistent and all the tests (unit, integration and acceptance) gave us green lights. A developer using a SNAPSHOT release, by definition, should be aware of the fact that what he/she is using is not final, but we can definitely make it clearer on our README.

  • somehow related to the point above, if we work in parallel on different branches (and different features), there is the risk to push intermittently SNAPSHOT releases with same version but quite different among them

This can indeed happen. Do you have any suggestion already to avoid such scenario ?
What if we create a snapshot artifact with the following convention:
com.truelayer:truelayer-java:$versionInGradlePropsFile-$branchName-SNAPSHOT ?
or even
com.truelayer:truelayer-java:$branchName-SNAPSHOT ?

This would lead to something like
com.truelayer:truelayer-java:0.4.10-jsdk-9-SNAPSHOT or com.truelayer:truelayer-java:jsdk-9-SNAPSHOT which would prevent pipeline on different branches pushing completely different stuff on the snapshot repo.

@tl-luca-baggi
Copy link
Contributor

tl-luca-baggi commented Mar 2, 2022

This would lead to something like com.truelayer:truelayer-java:0.4.10-jsdk-9-SNAPSHOT or com.truelayer:truelayer-java:jsdk-9-SNAPSHOT which would prevent pipeline on different branches pushing completely different stuff on the snapshot repo.

I think this would work fine, good idea 👍 I'd probably stick with the <version>-<jira-tag> to help devs understand on which stable release the unstable one is based on

@tl-andrea-dilisio
Copy link
Contributor Author

tl-andrea-dilisio commented Mar 3, 2022

This would lead to something like com.truelayer:truelayer-java:0.4.10-jsdk-9-SNAPSHOT or com.truelayer:truelayer-java:jsdk-9-SNAPSHOT which would prevent pipeline on different branches pushing completely different stuff on the snapshot repo.

I think this would work fine, good idea 👍 I'd probably stick with the <version>-<jira-tag> to help devs understand on which stable release the unstable one is based on

Implemented, this is the outcome on the snapshot repo!

Schermata 2022-03-03 alle 18 36 43

Finally, there's also another advantage in using a version like 0.4.10-jsdk-9-SNAPSHOT: if someone is using a snapshot version of the library we would find on logs more exact informations in the User-Agent HTTP header, as opposed to just using a $version-SNAPSHOT convention

@tl-andrea-dilisio tl-andrea-dilisio merged commit 9df6b68 into main Mar 4, 2022
@tl-andrea-dilisio tl-andrea-dilisio deleted the jsdk-9 branch March 4, 2022 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants