Remove jcenter, and oss.sonatype.org-releases repositories from public virtual repository; reconfigure Atlassian remote repositories

[MarkEWaite]

Service(s)

Artifactory

Summary

The data transfer volume from https://repo.jenkins-ci.org is still unexpectedly high. Log file analysis for a recent 5 day period shows that the top repositories based on the sum of the sizes of the requested files are:

Repository	Sum of GB requested
releases	2380
jcenter-cache	1787
incrementals	298
jcenter	98
nodejs-dist-cache	78
public	26
jgit-cache	14
oss.sonatype.org-releases-cache	8
snapshots	8
npm-dist-cache	5

JFrog has explained that even though we removed Apache Maven Central from the public repository, the contents of Apache Maven Central still exist as copies in the jcenter repository. We need to also remove the jcenter repository and the jcenter-cache repository from the definition of the public repository so that binaries from Apache Maven Central will be resolved from Apache Maven Central instead of being resolved from our repository.

When I had tried some initial experiments removing jcenter, several months ago, there were some cases where it appeared that we depend on artifacts that are only in jcenter and possibly only in our cache of jcenter. Unfortunately, my memory is weak and I cannot find any notes that reference that experiment.

I think that we should take the following steps:

• Create a new repository on repo.jenkins-ci.org and copy the org.connectbot.jbcrypt:jbcrypt:1.0.0 artifact from jcenter-cache to the new repository (required for Jenkins core, appears to only be available from jcenter-cache). Add the new repository to the public virtual repository
• Announce a 2 hour brownout where jcenter and jcenter-cache will be temporarily removed from the public repository
• Exclude the com.jayway.jsonpath groupId and all its packages from the jgit-cache. The artifactory plugin build failure and the robot plugin build failure will be resolved by that exclusion. The comment from @basil includes more details
• Add orphans repository to the public virtual repository
• Create a new remote repository corresponding to https://packages.atlassian.com/maven-public/ and replace the existing atlassian entry with the new one
• Define the tests we want to perform during the 2 hour brownout, including:
  • Build Jenkins core on ci.jenkins.io
  • Build Jenkins core on a freshly created virtual machine outside any CI infrastructure
  • Build 20+ popular plugins on ci.jenkins.io
  • Build 20+ popular plugins on a freshly created virtual machine outside any CI infrastructure
  • Build plugins that have shown issues (click links for analysis) including artifactory-plugin, blueocean-plugin, configuration-as-code-plugin, git-client-plugin, jira-plugin, promoted-builds-plugin, robot-plugin
  • Build plugins that require Java 8 and have shown issues including pipeline-aws-plugin, translation-plugin
  • Run the ci.jenkins.io weekly-test configuration of the plugin bill of materials
  • Build key Jenkins tools like the Maven HPI plugin and the plugin compatibility tester
• Prepare machines to test the brownout
  • Jenkins core build machine with Java 21 and Apache Maven 3.9.6 installed. Clone the Jenkins core repository to the machine but do not compile until the brownout has started
  • 5 Jenkins plugin build machines with Java 8, Java 11, Java 17, Java 21 and Apache Maven 3.9.6 installed. Clone the specific plugin repositories but do not compile until the brownout has started
• Run the brownout and perform the tests, collecting the results
  • Build Jenkins core on ci.jenkins.io
  • Build Jenkins core on a freshly created virtual machine outside any CI infrastructure
  • Build 20+ popular plugins on ci.jenkins.io
  • Build 20+ popular plugins on a freshly created virtual machine outside any CI infrastructure
  • Build plugins that have shown issues (click links for analysis) including artifactory-plugin, blueocean-plugin, configuration-as-code-plugin, git-client-plugin, jira-plugin, promoted-builds-plugin, robot-plugin
  • Build plugins that require Java 8 and have shown issues including pipeline-aws-plugin, translation-plugin
  • Run the ci.jenkins.io weekly-test configuration of the plugin bill of materials
  • Identify failures and investigate alternatives for those failures (for example, if we discover a library that is only available in jcenter, we can copy that library to another Jenkins repository)
• Reset the public configuration at the end of the brownout
• Implement the configuration change
• Request and analyze log files for the 5 day period after the configuration change
  • Confirm the jcenter-cache and jcenter repositories are no longer in the top 5 repositories

Reproduction steps

Steps that I took to analyze the requests:

1. Request Artifactory log files from JFrog
2. Update my artfiactory-sql fork to handle the new format of Artifactory log files
3. Upload the log files to a sqlite3 database
4. Identity top repositories with the SQL query
   select repo,sum(size_bytes)/1024/1024/1024 as SUM_GB from logs group by repo order by SUM_GB desc limit 10;

[MarkEWaite]

I built over 240 of the top 250 plugins with the settings.xml file listed later (using repo.jenkins-ci.org for the releases repository, the jgit repository, and the netbeans repository).

I had to download the jbcrypt library separately but other than that, I was able to build without running tests.

The repositories that did not build with those settings are:

• artifactory-plugin - cannot resolve com.jayway.jsonpath:json-path:jar:2.4.0 from token-macros dependency, works with my default settings (refer to analysis of failure)
• blueocean-plugin - cannot resolve com.atlassian.jira:jira-rest-java-client-api:jar:5.2.1, works with my default settings
• configuration-as-code-plugin - cannot resolve com.atlassian.crowd:crowd-integration-api:jar:5.1.5, works with my default settings (analysis of failure)
• jira-plugin - cannot resolve com.atlassian.jira:jira-rest-java-client-core:jar:5.2.7, works with my default settings (analysis of failure)
• promoted-builds-plugin - http downloads blocked for hudson.plugins:project-inheritance:jar:21.04.03, works with my default settings (analysis of failure)
• robot-plugin - cannot resolve com.jayway.jsonpath:json-path:jar:2.4.0, works with my default settings (analysis of failure)
• scp-plugin (fails to build even with my default settings and fails to build with Java 8 as well)

One plugin (gradle-plugin) does not build with Maven, so I can't check it.

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                              http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <pluginGroups>
    <pluginGroup>org.jenkins-ci.tools</pluginGroup>
  </pluginGroups>
  <profiles>
    <profile>
      <id>jenkins</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <repositories>
        <repository>
          <id>repo.jenkins-ci.org</id>
          <url>https://repo.jenkins-ci.org/releases/</url>
        </repository>
        <repository>
          <id>jgit.repo.jenkins-ci.org</id>
          <url>https://repo.jenkins-ci.org/jgit/</url>
        </repository>
        <repository>
          <id>netbeans.repo.jenkins-ci.org</id>
          <url>https://repo.jenkins-ci.org/netbeans/</url>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <id>repo.jenkins-ci.org</id>
          <url>https://repo.jenkins-ci.org/releases/</url>
        </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>
</settings>

[MarkEWaite]

@daniel-beck do you have a recommendation or a preference for the location where the org.connectbot.jbcrypt:jbcrypt:1.0.0 artifact from jcenter-cache should be placed? My first thought was to place it in the Jenkins releases repository because it is required for a Jenkins release and not available in our other repositories.

According to mvnrepository.com, that library is available from the Spring plugins repository and the Spring Lib M repository. The 1.0.1 and 1.0.2 versions of that library are available from repo.maven.apache.org, but the 1.0.0 version is not.

[timja]

or update whatever is using 1.0.0 to a newer version?

[daniel-beck]

My first thought was to place it in the Jenkins releases repository because it is required for a Jenkins release and not available in our other repositories.

That's an approach we've used in the past but would consider it mostly legacy from when just everything went into releases (there are issues complaining about differences in artifacts in releases vs. elsewhere, for example). Consider a new repo also added to the public virtual repo (populate exclusively by copying from cache to that new repo as admin).

[MarkEWaite]

or update whatever is using 1.0.0 to a newer version?

That would break builds of previous versions of Jenkins core, since it requires jbcrypt 1.0.0. I failed the last time that I tried to update the jbcrypt 1.0.0 dependency to a newer version. I don't have notes on the failure, but I remember that the attempt did not succeed.

I think it is a good idea to update the version of jbcrypt to a newer release if possible, but I think that should be handled separately from this effort.

[MarkEWaite]

Consider a new repo also added to the public virtual repo (populate exclusively by copying from cache to that new repo as admin).

Thanks. I've updated the text of the issue to propose the creation of a new repository, addition of that new repository to the public virtual repository, and the move of the jbcrypt 1.0.0 artifact into the new repository.

[dduportal]

This plan looks good to me, I'm gonna proceed with the first step. proposal for the new repository name: orphans to convey the "status" of the hosted artifacts within this repo.

[dduportal]

Create a new repository on repo.jenkins-ci.org and copy the org.connectbot.jbcrypt:jbcrypt:1.0.0 artifact from jcenter-cache to the new repository (required for Jenkins core, appears to only be available from jcenter-cache). Add the new repository to the public virtual repository

• A new "local" repository orphans has been created and added to the permission set "Anything" which grant read access to anyone (e.g. public access)
• The artifact org.connectbot.jbcrypt:jbcrypt:1.0.0 has been deployed to this new repository, along with its POM, sources and javadoc. A Maven metadat file has been autogenerated and looks the same as jcenter's
• Please note this repository is not included in the public virtual repository (yet?).

See: https://repo.jenkins-ci.org/artifactory/orphans/org/connectbot/jbcrypt/jbcrypt/

cc @MarkEWaite so you can continue your tests.

[MarkEWaite]

Analyzing artifactory plugin build failure

The artifactory plugin has a dependency on the token macro plugin 2.12. The token macro plugin 2.12 has a dependency on com.jayway.jsonpath:json-path:2.4.0. Our cached copy of the JGit repository includes an outdated copy of the json-path 2.4.0 pom file for the 2.4.0 release of json-path.

Based on comments in openhab/openhab-addons#10102 (comment) , we need to replace our outdated copy of 2.4.0 with an updated pom file and jar file.

If we don't want to replace the file in our cached copy of the JGit repository (because the outdated file still exists in the Eclipse JGit repository), we could place a fresh copy into the orphans repository and move the orphans repository before the JGit repository in our virtual repository search order.

Is there a way to safely remove or exclude the outdated / incorrect json-path 2.4.0 that we are getting from the Eclipse repository?

[MarkEWaite]

Analyzing configuration as code plugin build failure

The configuration as code plugin depends on the crowd2 plugin. The crowd2 plugin depends on the crowd-integration-client-common 5.1.5 library that appears to only be available from our jcenter-cache. Since crowd2 is an Atlassian product, I expected it to be in the atlassian-cache repository, but it does not seem to be there.

We could add the Atlassian repository to the configuration as code plugin like this:

diff --git a/pom.xml b/pom.xml
index f8c7d60e..9a1d5c37 100644
--- a/pom.xml
+++ b/pom.xml
@@ -51,6 +51,10 @@
       <id>repo.jenkins-ci.org</id>
       <url>https://repo.jenkins-ci.org/public/</url>
     </repository>
+    <repository>
+      <id>Atlassian</id>
+      <url>https://packages.atlassian.com/mvn/maven-atlassian-external/</url>
+    </repository>
   </repositories>
 
   <pluginRepositories>

That won't solve the case for other projects that depend on the Atlassian libraries. It also does not explain why the Atlassian libraries are not in our atlassian-cache repository but are in our jcenter-cache repository.

[MarkEWaite]

Analyzing Jira plugin build failure

The Jira plugin depends on Atlassian libraries that are available from our jcenter-cache but are not found unless jcenter-cache is included.

The build passed after I added the Atlassian repository to the jira-plugin pom file:

diff --git a/pom.xml b/pom.xml
index e3ca697..d56a0ff 100644
--- a/pom.xml
+++ b/pom.xml
@@ -330,6 +330,10 @@
       <id>repo.jenkins-ci.org</id>
       <url>https://repo.jenkins-ci.org/public/</url>
     </repository>
+    <repository>
+      <id>Atlassian</id>
+      <url>https://packages.atlassian.com/mvn/maven-atlassian-external</url>
+    </repository>
   </repositories>
 
   <pluginRepositories>

[MarkEWaite]

Analyzing robot plugin build failure

The robot plugin requires Java 11. It won't build with Java 21.

The robot plugin has a dependency on com.jayway.jsonpath:json-path:2.4.0. Our cached copy of the JGit repository includes an outdated copy of the json-path 2.4.0 pom file for the 2.4.0 release of json-path.

Based on comments in openhab/openhab-addons#10102 (comment) , we need to replace our outdated copy of 2.4.0 with an updated pom file and jar file.

If we don't want to replace the file in our cached copy of the JGit repository (because the outdated file still exists in the Eclipse JGit repository), we could place a fresh copy into the orphans repository and move the orphans repository before the JGit repository in our virtual repository search order.

Is there a way to safely remove or exclude the outdated / incorrect json-path 2.4.0 that we are getting from the Eclipse repository?

[MarkEWaite]

Analyzing promoted builds plugin build failure

The promoted builds plugin has an optional dependency on the project inheritance plugin 21.04.03. The project inheritance plugin 21.04.03 pom file uses http instead of https in its repository URL references. Maven rejects attempts to from an http repository.

There are hints like this on several pages that http blocking can be bypassed with a mirrorOf definition, but I've not been able to make it work.

[daniel-beck]

The artifact org.connectbot.jbcrypt:jbcrypt:1.0.0 has been deployed to this new repository, along with its POM, sources and javadoc. A Maven metadat file has been autogenerated and looks the same as jcenter's

Any particular reason why?

[MarkEWaite]

The artifact org.connectbot.jbcrypt:jbcrypt:1.0.0 has been deployed to this new repository, along with its POM, sources and javadoc. A Maven metadat file has been autogenerated and looks the same as jcenter's

Any particular reason why?

I could not find the artifact org.connectbot.jbcrypt:jbcrypt:1.0.0 on repo.jenkins-ci.org in any repository other than jcenter-cache. Since the goal is to remove jcenter-cache from the definition of the public virtual repository, it seemed like we need to copy that artifact to a new repository that can be included in the public virtual repository without including all of jcenter-cache.

Did that address your question or is there something more that I'm not understanding in the question?

[daniel-beck]

Did that address your question or is there something more that I'm not understanding in the question?

From my proposal of the new repo holding these dependencies:

populate exclusively by copying from cache to that new repo as admin

That trivially ensures identical artifacts. Per Damien,

has been deployed to this new repository, along with its POM, sources and javadoc. A Maven metadat file has been autogenerated and looks the same as jcenter's

That looks very manual and results in differences. For example, we don't have *.asc files now. No idea whether they're needed, but any difference seems unnecessary.

If my proposed approach is not possible, or has drawbacks I'm unaware of, and that's the reason for the different approach, I'd like to know.

[basil]

Is there a way to safely remove or exclude the outdated / incorrect json-path 2.4.0 that we are getting from the Eclipse repository?

Googling brings up https://jfrog.com/help/r/jfrog-artifactory-documentation/basic-settings-for-remote-repositories which shows that you can configure exclude patterns on a remote repository (i.e., a cache). Based on this documentation I think we should delete the json-path dependency from our Eclipse remote repository (i.e., our cache) and add an exclude pattern so that it won't be downloaded again. Then requests for it from the JGit remote repository should 404 and the lookup should proceed to the (valid) version on Central.

[basil]

Confluence Publisher is also impacted: #3856

[dduportal]

The artifacts are there in the remote repositories:

* https://packages.atlassian.com/maven-public/com/atlassian/jira/jira-rest-java-client-core/5.2.7/

* https://packages.atlassian.com/mvn/maven-atlassian-external/com/atlassian/jira/jira-rest-java-client-core/5.2.7/

https://repo.jenkins-ci.org/atlassian-maven-public/ is returning 401, not 404. Seems like it was not set up for anonymous access (i.e., an unrealistic test environment).

Pairing operation with @MarkEWaite today: the repository https://repo.jenkins-ci.org/artifactory/atlassian-maven-public is now available publicly (added to the permissions scheme "Any Remote" which adds Read and Cache to the anonymous user.

[dduportal]

Start of the 2nd brownout on repo.jenkins-ci.org with @MarkEWaite:

We've updated the public virtual repository to the following setup:

curl --silent https://repo.jenkins-ci.org/api/repositories/public | jq -r '.'
{
  "key": "public",
  "packageType": "maven",
  "description": "All public repositories aggregated for the purpose of building Jenkins and its plugins",
  "repositories": [
    "releases",
    "orphans",
    "atlassian-maven-public",
    "shibboleth",
    "maven.jenkins-ci.org",
    "maven.twttr.com",
    "javanet2",
    "tmate",
    "cloudbees.public.release",
    "jgit",
    "netbeans",
    "fusesource",
    "snapshots"
  ],
  "rclass": "virtual"
}

• Removed jcenter and atlassian
• Added orphans and atlassian-maven-public

[MarkEWaite]

After implementing the production change, we found that the artifactory plugin depends on a 3.1.7 release of a platform pom that is only available from the Spring plugins repository at https://repo.spring.io/plugins-release/ . Other versions of that artifact are available from Atlassian public, but not that version.

@dduportal created a jcenter-orphans remote repository that includes a specific item

com.atlassian.pom:public-pom:pom:4.0.7 was then missing. That was fixed by adding another entry to jcenter-orphans

com.atlassian.sal:sal-api:jar:3.0.7 was missing.

Items that were missing have been added to the jcenter-orphans remote repository with a narrow inclusion definition to avoid including any more of jcenter than those few items that are not available elsewhere. The jcenter-orphans repository is intentionally placed near the end of the search list so that the official Atlassian repository cache is used first.,

[MarkEWaite]

The crawler job that detects updated tools is failing on ci.jenkins.io with a Grape error

• https://ci.jenkins.io/job/Infra/job/crawler/job/master

Resolved by adding Maven central to the Grape repository list in:

• Add Maven Central to Grape configuration crawler#134

[dduportal]

The crawler job that detects updated tools is failing on ci.jenkins.io with a Grape error

* https://ci.jenkins.io/job/Infra/job/crawler/job/master

Resolved by adding Maven central to the Grape repository list in:

* [Add Maven Central to Grape configuration crawler#134](https://github.com/jenkins-infra/crawler/pull/134)

Thanks @basil for the batch of PRs to fix and improve the crawler (and @MarkEWaite @timja @NotMyFault for the revies/deploys).

[dduportal]

Update:

• Update the allow list pattern of jcenter to solve declarative-pipeline-migration-assistant-plugin no longer compiles #3872 (see issue for details)
• Removed jcenter-cache and atlassian-cache from both Any Remote and Anything permissions scheme to ensure they cannot be reached anonymously (as such the outbound bandwidth should drastically decrease)

[MarkEWaite]

• Removed jcenter-cache and atlassian-cache from both Any Remote and Anything permissions scheme to ensure they cannot be reached anonymously (as such the outbound bandwidth should drastically decrease)

Thanks very much. Should the same change be applied to oss.sonatype.org-releases since it also contains a copy of Apache Maven Central?

[MarkEWaite]

Jenkins core 2.439 and later now depend on a version of the jbcrypt library that is available from Apache Maven Central. Thanks to @andham for that improvement. Refer to the merged pull request:

• [JENKINS-72466] Upgrade jbcrypt dependency jenkinsci/jenkins#8811

[dduportal]

• Removed jcenter-cache and atlassian-cache from both Any Remote and Anything permissions scheme to ensure they cannot be reached anonymously (as such the outbound bandwidth should drastically decrease)

Thanks very much. Should the same change be applied to oss.sonatype.org-releases since it also contains a copy of Apache Maven Central?

The repository oss.sonatype.org-releases is no longer available anonymously: it requires authentication.

[dduportal]

Update:

• The repository orphans has been removed (in favor of adding filters in the jcenter-orphans mirror repository - Remove orphans local repository #3884
• We forgot to clean the ACP cache: Artifact caching proxy repo.do.jenkins.io is not answering requests from bom build #3889