Builds are often failing with timeout due to Artifactory being slow

[timja]

Service

ci.jenkins.io

Summary

A plugin with no Java code which should build in ~1min or less is still building after 30 mins

jenkinsci/jenkins-infra-test-plugin#16
https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fjenkins-infra-test-plugin/detail/PR-16/3/pipeline/

Reproduction steps

No response

[timja]

Seems to affect windows as well:
https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fsshd-plugin/detail/PR-50/1/pipeline/30

[jglick]

Smells like an Artifactory issue. https://issues.apache.org/jira/browse/MNG-6976 as usual.

[dduportal]

• Confirmled that the download speed is the culprit: https://ci.jenkins.io/job/Plugins/job/jenkins-infra-test-plugin/job/PR-16/8/console
• I can reproduce on my local ISP (Belgium)
• We opened an issue at JFrog support: https://groups.google.com/g/jenkins-infra/c/laSsgPOH9qs

Thanks @jglick @timja for reporting!

[jmMeessen]

My case of build slowness: a full CI of config-file-provider (https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fconfig-file-provider-plugin/activity) takes about 30 minutes.

It now fails (after timeout exhaustion) after 1 hour. See for example https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fconfig-file-provider-plugin/detail/master/169/pipeline

[timja]

Seems fixed now, took 2 minutes instead of 25min +

https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fjenkins-infra-test-plugin/detail/PR-16/10/pipeline

[lemeurherve]

There seems to be some issues again, reopening.

[jmMeessen]

It looks like we are experiencing a similar disruption: build take much more than usual to build. While it usually takes max 30 mins, it is still building at 53 minutes. https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fconfig-file-provider-plugin/detail/master/172/pipeline/

Did not do a thorough analysis to confirm that the cause (slow artifactory link) is the same

[lemeurherve]

* We opened an issue at JFrog support: [groups.google.com/g/jenkins-infra/c/laSsgPOH9qs](https://groups.google.com/g/jenkins-infra/c/laSsgPOH9qs)

Here is the thread on jenkins-infra mailing list: https://groups.google.com/g/jenkins-infra/c/j7M49evRXJg

[reinholdfuereder]

Today I started stumbling over (I think) this problem as well (in the course of updating Jenkins core 2.319.1 to 2.319.2 and a few plugins) in a Gradle build for my shared Jenkins pipeline library:
More or less suddenly it started to become very slow.
With the quite surprising extremely helpful Gradle Build Scan (free!) one can see:

[jmMeessen]

@dduportal @lemeurherve FYI:
I started a build on config-file-provider plugin beginning of the EU morning and it worked : no timeout/kill after 1 hour. The build took the usual 30 minutes.
I merged another PR right now. Let's see if the good news is confirmed.

[dduportal]

We just got a feedback from JFrog in https://groups.google.com/g/jenkins-infra/c/j7M49evRXJg/m/9FG7uoQkAQAJ .

Are your builds speed improving?

[jmMeessen]

I am sad to say that the last build was shot down after 1 hour: https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fconfig-file-provider-plugin/detail/master/177/pipeline

[jmMeessen]

Change was concurrent. "BONUS: Same player shoots again"

[timja]

This also failed with the same issue: jenkinsci/azure-container-agents-plugin#97
(1 hour ago)

and Reinhold yes same issue, that is a nice feature of gradle 😄

[timja]

JFrog just upgraded our instance and it works for me on my machine now fine.

[timja]

Seems fixed to me now, the build above that was failing now works

[dduportal]

Now confirmed that it has been fixed by the awesome people at Jfrog, either from local builds and on the jenkins-infra.

Closing the issue.

Please feel free to reopen if you get any slowness again (by explaining where you are located and giving more details to help us and JFrog to diagnose).

[reinholdfuereder]

Unfortunately it is happening again:
Please note that this is a clean build (without any Gradlew cache whatsoever due to a clean Docker container and no Docker volume), so all dependencies must be fetched again and again.
And I triggered this build a few times this morning (i.e in the previous 3.5h; now it is ~10:30 Europe/Vienna timezone):

• still OK at ~08:00
• slow since at least ~09:45 (but not extremely slow, just significantly slower)
  • E.g.
    

[lemeurherve]

Back to an average of 5Kb/s download rate too.

[dduportal]

Pinged Jfrog: https://groups.google.com/g/jenkins-infra/c/j7M49evRXJg/m/JREWgDJvAQAJ

[MarkEWaite]

Reported as resolved in the email thread at https://groups.google.com/g/jenkins-infra/c/j7M49evRXJg/m/3UhhLNl9AQAJ