[ci] publish snapshots with every commit to main

[Shastick]

Adds a workflow for publishing a snapshot to sonatype with each commit to main.

[langchain4j]

@Shastick could you please provide a bit of details, I am not sure I understand how it works and why we need this. Thanks a lot!

[Shastick]

This was meant as a draft to try out publishing to sonatype via github actions. I'll reopen when I had some time to explore this

[nooone]

This would actually be pretty useful, especially while this is still under active development. Would be nice to get bugfixes and new features faster than once every few weeks with a new release.

[Shastick]

@langchain4j should I take a look at this again?

I’ll likely need some credentials to push to Sonatype:

You could either grant me access (I have a sonatype account that I could give you the name off), or add your credentials as secrets somewhere in the Github actions settings.

[langchain4j]

@Shastick where can/should we host snapshots? By Sonatype you mean the same repo where we host released artefacts?

[Shastick]

@Shastick where can/should we host snapshots? By Sonatype you mean the same repo where we host released artefacts?

Yes, at sonatype: if I remember correctly the process is roughly the same as for a normal release, except that:

• you'll push to the snapshot repository (plugins usually do this automatically based on the version)
• there is no need to manually promote the artifact (again, if I remember correctly)

[langchain4j]

Hi @Shastick, sorry for the delay. I have added OSSRH_USERNAME and OSSRH_PASSWORD repository secrets, I guess this should be enough to publish snapshots?

[Shastick]

Hi @Shastick, sorry for the delay. I have added OSSRH_USERNAME and OSSRH_PASSWORD repository secrets, I guess this should be enough to publish snapshots?

I'm giving this a try and seem to get 401's from sonatype: are the credentials still up to date?

Ie, we get:

Failed to execute goal org.sonatype.plugins:nexus-staging-maven-plugin:1.6.13:deploy (injected-nexus-deploy) on project ***-document-parser-apache-poi: Failed to deploy artifacts: Could not transfer artifact dev.***:***-core:jar:javadoc:0.26.0-20240119.094028-2 from/to ossrh (https://s01.oss.sonatype.org/content/repositories/snapshots): authentication failed for https://s01.oss.sonatype.org/content/repositories/snapshots/dev/***/***-core/0.26.0-SNAPSHOT/***-core-0.26.0-20240119.094028-2-javadoc.jar, status: 401 Unauthorized -> [Help 1]

For example on this run

@langchain4j could you try running a snapshot build locally (I assume it's a mvn deploy or mvn --batch-mode deploy on the master branch without changing any version) with the credentials set in MAVEN_USERNAME and MAVEN_PASSWORD and report if it works or not?

[langchain4j]

Hi @Shastick sorry for delay.

I have set MAVEN_USERNAME and MAVEN_PASSWORD env variables locally and run mvn clean deploy -DskipTests on current master, it works fine: https://s01.oss.sonatype.org/content/repositories/snapshots/dev/langchain4j/langchain4j-core/0.26.0-SNAPSHOT/

I am not sure the need for MAVEN_USERNAME and MAVEN_PASSWORD as I have user/password configured in .m2/settings.xml.

I have updated OSSRH_PASSWORD and OSSRH_TOKEN with same values as in my .m2/settings.xml just in case.

[langchain4j]

Maybe try using OSSRH_PASSWORD and OSSRH_TOKEN names for env variables? This worked fine for me: https://github.com/ai-for-java/openai4j/blob/main/.github/workflows/release.yml

[Shastick]

@langchain4j Looks like we make it one step further, but now:

Deployment failed: repository element was not specified in the POM inside distributionManagement element or in -DaltDeploymentRepository=id::layout::url parameter -> [Help 1]

I'll try to check later, but possibly you have something in your settings about distributionManagement that we could add to the aggregator pom?

[langchain4j]

@Shastick I am getting the same error locally. I guess it tries to deploy aggregator pom which does not contain all mandatory information like scm links, etc. But we do not publish anyway, so we could exclude this from the build. Other modules are published ok as far as I see, right?

BTW could you please remind why we went with this parent/aggregator separation?

[Shastick]

Looks like snapshot publishing was successful, I'll update the action so it only runs on master and this should be ready to go.

[langchain4j]

might use matrix approach as we do in https://github.com/langchain4j/langchain4j/blob/main/.github/workflows/main.yaml

[Shastick]

Indeed that would make sense.

What is the approach for releases? Is everything released from the older version of Java that is possible?

If you have a script or a bunch of options that are used for release they could be useful here, otherwise I can cobble something together from the existing workflow files.

[langchain4j]

What is the approach for releases? Is everything released from the older version of Java that is possible?

Yes,

• we release with java 8 everything except langchain4j-opensearch, langchain4j-neo4j and langchain4j-code-execution-engine-graalvm-polyglot
• then langchain4j-opensearch with 11
• then langchain4j-neo4j and langchain4j-code-execution-engine-graalvm-polyglot with 17

I do not have any script unfortunately.

[Shastick]

I'll have a look, I think that should be feasible in some way

[Shastick]

One side question though: is it important that all snapshots for a release from master share the same version?

Currently it seems based on time: I assume we could have something based on the commit hash if required.

[langchain4j]

Yes, I guess users should be able to use X.Y.Z-SNAPSHOT version and maven should provide the latest snapshot, or?

[Shastick]

Ok, makes sense. I'll look into that once I got the rest to work

[langchain4j]

@Shastick thanks a lot! What was the problem BTW? wrong env variable names?

[Shastick]

@Shastick thanks a lot! What was the problem BTW? wrong env variable names?

Yes, it seems I got confused with the environment variables

[Shastick]

@langchain4j did we maybe overthink things a bit? Given the maven.compiler.target options in the pom files, I believe that building and releasing everything from a recent version of java should still produce jars that can be consumed by an older JVM where possible?

[Shastick]

Release works (see this run for example).

I assume we can trust the backward compatibility (all modules specify the minimal java version to target), so we don't need to build on multiple JDKs (and we can always revisit if required)

Also, it seems that the version is as expected: ie, here is the metadata for langchain4j-neo4j:0.26.0-SNAPSHOT. Multiple releases of the same snapshot version can be done.

All modules can be seen here: https://s01.oss.sonatype.org/content/repositories/snapshots/dev/langchain4j/

[langchain4j]

@Shastick thank you so much and sorry for the late reply. Yeah seems that I indeed overthinked it and we can have a single build. Awesome!

[langchain4j]

@Shastick great job, thank you!