SNAPSHOT release link for MacOs is broken

[baditaflorin]

https://oss.sonatype.org/service/local/artifact/maven/content?r=snapshots&g=org.openrefine&a=openrefine&v=3.8-SNAPSHOT&c=mac&p=dmg
in the readme file

Also here
https://openrefine.org/docs/manual/installing

the link is broken
https://github.com/OpenRefine/OpenRefine-snapshot-releases/releases

[tfmorris]

There are couple of different issues here:

• Snapshot releases aren't being uploaded (for all the platforms, not just Mac). I'm not sure why they were switched to 3.8-SNAPSHOT in December when we released 3.7.2 in April. Seems like they should still be 3.7-SNAPSHOT, but that's not the issue.
• Documentation refers to a snapshot repo which has been deleted

It looks from the build logs like the upload steps are getting skipped:

[tfmorris]

Not sure if it has anything to do with this, but I also stumbled across this strange Github user account: https://github.com/openrefine-snapshot-uploader

[wetneb]

Snapshot releases aren't being uploaded (for all the platforms, not just Mac). I'm not sure why they were switched to 3.8-SNAPSHOT in December when we released 3.7.2 in April. Seems like they should still be 3.7-SNAPSHOT, but that's not the issue.

That's intentional: the 3.7-SNAPSHOT version is only appropriate for the 3.7 branch, not the master branch (out of which we are yet to make the first release of the 3.8 series).

Documentation refers to a snapshot repo which has been deleted

That's obviously a problem, it should instead point to https://github.com/OpenRefine/OpenRefine#snapshot-releases

Not sure if it has anything to do with this, but I also stumbled across this strange Github user account: https://github.com/openrefine-snapshot-uploader

This was previously used to upload releases to the dedicated repository where we stored snapshot releases. Let me delete it.

[wetneb]

The upload is currently failing because Apple refuses to notarize our package:

A required agreement is missing or has expired. This request requires an in-effect agreement that has not been signed or has expired. Ensure your team has signed the necessary legal agreements and that they are not expired.

I will do the necessary to get this signed by our fiscal sponsor and that should fix the upload of snapshots.

[wetneb]

This actually seems to be a problem in Maven Central. I checked Gephi's snapshot releases and they also raise a 404 error:
https://github.com/gephi/gephi#latest-releases

[wetneb]

I submitted a report here: https://issues.sonatype.org/projects/MVNCENTRAL/issues/MVNCENTRAL-8165?filter=allissues

[tfmorris]

I'm not sure what's going on, but clicking that link brings up MVNCENTRAL-8163 instead of the desired ticket. I tried logging in as well, but it didn't change the behavior.

[wetneb]

Yes it's also broken for me. Does this link work any better?
https://issues.sonatype.org/browse/MVNCENTRAL-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedId=1279187#comment-1279187

[tfmorris]

That one says "You can't view this issue. It may have been deleted or you don't have permission to view it." I'm logged in, but the only group I'm part of is jira-users when I was previously a member of the org.openrefine group, I think.

[wetneb]

It seems that it is only visible to "Reporter and Sonatype", as if it were a security vulnerability of some sort, so it probably has not got anything to do with group membership. I will post a copy of the thread there once it is settled.

[wetneb]

Here are the logs of this conversation:

Me:

We have noticed that the API we use to download snapshot releases published to Maven Central has started failing.

Here are examples of how we call the API:

• https://oss.sonatype.org/service/local/artifact/maven/content?r=snapshots&g=org.openrefine&a=openrefine&v=3.8-SNAPSHOT&c=linux&p=tar.gz
• https://oss.sonatype.org/service/local/artifact/maven/content?r=snapshots&g=org.gephi&a=gephi&v=0.10.2-SNAPSHOT&c=linux-x64&p=tar.gz

Those links fail, with errors like this:

Path /org/openrefine/openrefine/3.8-SNAPSHOT/openrefine-3.8-20230126.100009-68-linux.tar.gz not found in local storage of repository "Snapshots" [id=snapshots]

What is curious is that the filename mentioned in this message is a very old one: I am pretty sure we have published other snapshots since this date.

The fact that this problem occurs both for OpenRefine and Gephi, two independent projects (both of which publish snapshots in their CI/CD system), makes me think that this is likely a problem in Maven Central itself.

Sonatype support:

Snapshot hosting on oss.sonatype.org, while it is a service that has been available to the publishing community since oss.sonatype.org's inception isn't officially supported (vs. support for using the staging workflow to publish to the Releases repository, which, in turn, is the source for artifacts that are synced to Maven Central, i.e. repo1.maven.org).

I can trigger a reindex of the Snapshots repository for this particular component, but I'd like to understand a little bit more of your use case for using the API to download SNAPSHOTs (vs. setting up oss.sonatype.org as a Snapshot repository in your Maven configuration).

My reply:

Thanks for this insight!

We use this API to offer download links of packaged artifacts for our end users. Those artifacts contain a ready-to-use desktop application that people can just install. Therefore we cannot expect users (who are not programmers) to install Maven and write a pom.xml file to download the artifacts.

But perhaps this is not an appropriate use of the repository and we should rather be storing such files elsewhere?

Sonatype support:

I would worry about telling you users to rely on SNAPSHOT versions to begin with since those are meant to be ephemeral or superseded as soon as you publish a release version. But, more practically, while there are no immediate plans to stop supporting the ability for users to publish their SNAPSHOT versions to oss.sonatype.org, given that SNAPSHOT storage is not primary functionality on that host, it's possible that we will cease to offer that functionality in the future (e.g. if it proves to be too expensive to support).

My reply:

Ok!

Most of our users rely on our stable versions instead (which can be downloaded from other mirrors) - the SNAPSHOT versions are there to offer previews from the development version, for debugging purposes. So it does not seem to a be a misuse of the principle of SNAPSHOT versions to me.

If access to the SNAPSHOT releases via this API is not officially supported, then we need to migrate to something else.

Would there be any other API on another host that would let us access SNAPSHOT artifacts?

Sonatype support:

Sonatype does not officially support SNAPSHOT hosting on any of the hosts it runs to support Maven Central publishers. Sonatype does offer an open-source version of Nexus Repository Manager if you wish to self host a repository manager that can serve as your Snapshot repository.

[wetneb]

Given that the way we are using the snapshot hosting does not seem supported, I think we should migrate to some other system. One first thing we could do, as a temporary measure, is to store the snapshot releases as job artifacts on the GitHub Actions jobs that create them. That is not going to be as convenient as the current situation since we won't have (I think) an easy way to link to the latest snapshot, but at least it would help us point users to a particular build.

Another solution would be to upload the artifacts to some Netlify site that we would serve as a subdomain of openrefine.org, but I suspect we might run into issues with maximum file sizes.

Any other hosting solutions we should consider?

[thadguidry]

At Ericsson we used Jfrog and never a problem. You'll likely need a Bintray account also but that might have changed. See the list at bottom of Apache article https://maven.apache.org/repository-management.html

[wetneb]

That list is super helpful, I did not realize that so many platforms supported the protocol used by Apache Maven.
Looking at the JFrog documentation I can only see references of self-hosted instances - is there any public service with free access for FOSS projects?

[thadguidry]

You'll have to reach out to their support and I'd encourage to do that first. The software is free for the level of integration and options that we need. So only hosting is needed which means acquiring free credits with either Microsoft Azure, GCP, AWS, Oracle clouds. Where many do offer free credits for FOSS projects so I'd check with CS&S or others.

[wetneb]

hmm I am not sure about getting free credits for a cloud hosting provider: that amounts to us self-hosting the tool, and the maintenance burden feels intuitively way too high for the feature we want to get out of it.

Since #5833 was merged, the snapshot releases are available as workflow artifacts. That's a little bit better, but still not good enough because one needs to be logged in with a GitHub account to download them, and also because I don't see any way to make a permanent link to the latest version of each of those artifacts.

[thadguidry]

I can volunteer to maintain and run an Artifactory instance if it helps. But maybe indeed it's easier to just let folks get a GitHub login if they wish to help try and test our snapshots. The snapshots also could be copied to somewhere in GitHub where they could download without the login hassle perhaps with GitHub Packages (which is Maven run in Azure cloud by Microsoft anyways)?

<repository>
          <id>github</id>
          <url>https://maven.pkg.github.com/OWNER/REPOSITORY</url>
          <snapshots>
            <enabled>true</enabled>
          </snapshots>
        </repository>

https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-apache-maven-registry

[wetneb]

Nice, I did not realize that this was available.
The problem is that there does not seem to be any API available to download those packages using a predictable URL. See this investigation:
https://josh-ops.com/posts/github-download-from-github-packages/

Also, packages uploaded to this repository can only be downloaded while being logged in to GitHub, which means we have essentially the same restrictions as the current GitHub Actions artifacts.

For this reason I am proposing to just update the README.md file to teach users how to download those artifacts themselves: #5972.