Mirrors of jenkins update is not working

[ayanamist]

Service

Update center

Summary

I'm a user in china, and i see there are mirrors in china from http://mirrors.jenkins.io/status.html and https://updates.jenkins.io/download/war/latest/jenkins.war?mirrorstats
However, since Jenkins Updates Sites is a Version-Specific Update Sites which requires version parameter to be handled, all mirrors do not implement this, and must be set to version-locked-in site url like https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/dynamic-stable-2.319.3/ but not https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates
What's worse is that, plugin.url in update-center.json are always pointed to updates.jenkins.io, no matter what site i choosed, jenkins will always download from updates.jenkins.io which is very very slow in china. And update-center.json can't be rewritten since it's protected by certificate verification.
Is there any other way to use these mirrors to accelerate downloading and decrease the network load of main site?

Reproduction steps

No response

[dduportal]

Hi @ayanamist thanks for reporting!
I'm not 100% sure of what I say, so please bear with me if I'm mistaken.

But the update-center.json is not mirrored, as you underlined, to ensure that its safety.

However, all the "heavy" files (plugins, war, packages) are mirrored: each call for these files on the domain updates.jenkins.io are redirected at one point to get.jenkins.io which is the mirror system that would redirect you to the mirror close to you (for the war/hpi/jpi/etc.).

If I understand correctly the problem you are facing, it means that you see slow answers for the HTTP/302 redirect on update.jenkins.io: is that correct?

Could you share some data around timing of theses requests (and their redirect chain) as seen on your side, to help us understand better and search for solutions?

[ayanamist]

update-center.json is not a small file, and we have lots of machines inside Alibaba Cloud without internet access (to improve security), we access these machines via a VPN gateway. Alibaba Cloud also provides a jenkins mirror which is not listed https://mirrors.aliyun.com/jenkins/ and can be accessed inside VPC without internet (through special domain https://mirrors-ssl.aliyuncs.com/jenkins/ )
Is there any way to make current mirror system handle this intranet situation?

[daniel-beck]

What's worse is that, plugin.url in update-center.json are always pointed to updates.jenkins.io, no matter what site i choosed, jenkins will always download from updates.jenkins.io which is very very slow in china. And update-center.json can't be rewritten since it's protected by certificate verification.

updates.jenkins.io should redirect to get.jenkins.io which is mirrored. Is even the redirect slow?

But the update-center.json is not mirrored, as you underlined, to ensure that its safety.

Actually, that is probably doable, if we tolerate some delays in delivering metadata to users. This is how it has worked for a really long time in the past, that went away progressively while switching to serving only HTTPS. In general the current solution is better IMO, should look at other solutions first.

---

So, options.

Individual orgs could package Jenkins with their own custom root certs to support self-hosted local/regional update sites.

Alternatively, why can't a mirror be set up that has the same mod_rewrite rules for ?version as the full site? Can probably be a different entrypoint to keep the actual mirror host simple, but that would then redirect callers to the mirror. https://github.com/jenkins-infra/update-center2/blob/master/site/generate-htaccess.sh is the script generating the htaccess file we use on updates.jenkins.io. What's left is then the bounce from updates.jenkins.io to get.jenkins.io to your local mirror for each individual file being downloaded.

[ayanamist]

Alternatively, why can't a mirror be set up that has the same mod_rewrite rules for ?version as the full site?

Because all mirrors listed in http://mirrors.jenkins.io/status.html does not support this, because today apache httpd is not widely used anymore so htaccess is not usable.

updates.jenkins.io should redirect to get.jenkins.io which is mirrored. Is even the redirect slow?

Visit https://updates.jenkins.io/update-center.json?version=2.319.3 will be redirected to https://updates.jenkins.io/dynamic-stable-2.319.3/update-center.json which will output a huge json directly, and this output is unstable in China, sometimes will timeout.

[daniel-beck]

Alternatively, why can't a mirror be set up that has the same mod_rewrite rules for ?version as the full site?

Because all mirrors listed in http://mirrors.jenkins.io/status.html does not support this, because today apache httpd is not widely used anymore so htaccess is not usable.

Yes. I know it does not exist. I am trying to understand what prevents the creation, if anything. I even mention it can be a separate system, so the "nobody uses Apache" reason doesn't really apply, unless it's not usable.

updates.jenkins.io should redirect to get.jenkins.io which is mirrored. Is even the redirect slow?

Visit https://updates.jenkins.io/update-center.json?version=2.319.3 will be redirected to https://updates.jenkins.io/dynamic-stable-2.319.3/update-center.json which will output a huge json directly, and this output is unstable in China, sometimes will timeout.

I am referring to the download URLs listed in the JSON, e.g.

$ curl -IL https://updates.jenkins.io/download/plugins/matrix-auth/3.1/matrix-auth.hpi
HTTP/1.1 302 Found
Date: Wed, 02 Mar 2022 09:18:03 GMT
Server: Apache/2.4.29 (Ubuntu)
Location: https://get.jenkins.io/plugins/matrix-auth/3.1/matrix-auth.hpi
Content-Type: text/html; charset=iso-8859-1

HTTP/2 302 
date: Wed, 02 Mar 2022 09:18:04 GMT
content-type: text/html; charset=utf-8
location: https://ftp.halifax.rwth-aachen.de/jenkins/plugins/matrix-auth/3.1/matrix-auth.hpi
cache-control: private, no-cache
link: <https://ftp.belnet.be/mirror/jenkins/plugins/matrix-auth/3.1/matrix-auth.hpi>; rel=duplicate; pri=1; geo=be
strict-transport-security: max-age=86400; includeSubDomains; preload

HTTP/2 200 
server: nginx/1.18.0
date: Wed, 02 Mar 2022 09:18:04 GMT
content-type: application/octet-stream
content-length: 242148
last-modified: Tue, 01 Mar 2022 10:52:28 GMT
accept-ranges: bytes

[dduportal]

Hello @ayanamist , do you have any news on this issue?

[ayanamist]

I am trying to build a internal repo in my corp, but as i said before, no one knows how to maintain an apache httpd instance, and lack of docs in https://github.com/jenkins-infra/update-center2 and how to put our own CA certs in jenkins k8s pod, which are main difficulties. So no more news.

[daniel-beck]

lack of docs in https://github.com/jenkins-infra/update-center2

What specifically is missing?

how to put our own CA certs in jenkins k8s pod

Put files in $JENKINS_HOME/update-center-rootCAs per https://github.com/jenkinsci/jenkins/blob/c46f96866d54f851731e5498118ddd3b5b61aa5f/core/src/main/java/jenkins/util/JSONSignatureValidator.java#L286.

[ayanamist]

@daniel-beck

lack of docs in https://github.com/jenkins-infra/update-center2

As first line of README "This project is primarily used to generate the jenkins.io update center layout.", so if you follow the guide in the README, it won't get a working private update site, maybe there are many implicit prerequisites i dont know.

how to put our own CA certs in jenkins k8s pod
Put files in $JENKINS_HOME/update-center-rootCAs per

It's not as easy as you said to put this cert into k8s pod, if insisting on using official jenkins image, it must declare a volume and use an init container to put file to the volume, and add a environment variable to use this new update site by default without user modification.

I hope you can give some further practical advices but not these meaningless words.

[dduportal]

@daniel-beck

lack of docs in https://github.com/jenkins-infra/update-center2

As first line of README "This project is primarily used to generate the jenkins.io update center layout.", so if you follow the guide in the README, it won't get a working private update site, maybe there are many implicit prerequisites i dont know.

how to put our own CA certs in jenkins k8s pod
Put files in $JENKINS_HOME/update-center-rootCAs per

It's not as easy as you said to put this cert into k8s pod, if insisting on using official jenkins image, it must declare a volume and use an init container to put file to the volume, and add a environment variable to use this new update site by default without user modification.

I hope you can give some further practical advices but not these meaningless words.

Hi @ayanamist , we understand the frustration it causes in your case but please can you watch your tone?

This is a community space and a lot of the work done here is done voluntary. The code here is aimed at the principal update center and has been written like this for reasons.

If you feel that the documentation is missing, you can open a PR proposing improved content. It also implies that maintainer have the right to say "no".

Daniel is trying to help, as I am but we are not entitled to solve issues of every people asking as we are not a business provider.

[ayanamist]

@dduportal I know here is just a community, but i do not like someone quote my limited words and given some meaningless solution as saying "How to put an elephant into a fridge is as easy as open fridge, put it into it and close the door", yeah that's a solution, but is it useful, or just say nothing?

[daniel-beck]

As first line of README "This project is primarily used to generate the jenkins.io update center layout.", so if you follow the guide in the README, it won't get a working private update site, maybe there are many implicit prerequisites i dont know.

You haven't responded to my question (that I asked so that I can improve the documentation); and also seem to ignore the very next sentence from the readme. It's not that it cannot be used outside Jenkins project infra, I know of some orgs who use it for their own update sites. In fact, the separation of site/generate.sh and site/publish.sh exists to allow that by splitting the generic behavior from the Jenkins infra specific behavior. It's just to make sure expectations are realistic; for example we've rejected PRs adding command-line options we would not use.

I am aware that we do not provide a solution that works "out of the box" but if all suggestions are met with "doesn't work with our stack" (and the implied "and we're unwilling to change anything about it"), I don't think we'll get anywhere here.

[ayanamist]

The original purpose of this issue is just recalling some jenkins people to reconsider the json signature and direct download url which bring troubles for mirroring.
Other issues related with my own stack will get done by myself, and i describe them just to notice you guys what mirroring troubles are, there are many similar problems in chinese community aming lots of blog articles.
#2787 (comment) is just describing the difficulties for workarounds but not means i cant handle them.
Now i just heard the answers from you. That's all.

[daniel-beck]

Good point, the original request (having a good download experience in China) is reasonable. I lost track of it after offering what I thought were reasonable workarounds.

reconsider the json signature and direct download url which bring troubles for mirroring

JSON signature will need to remain in some manner to ensure the integrity of the downloads. There could be potential improvements in Jenkins to work around that, like support rewriting URLs from one host to another, without compromising integrity.