Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ftp.belnet.be should be removed from mirrors or a fall-back offered #3784

Closed
kwisatz opened this issue Oct 15, 2023 · 13 comments
Closed

ftp.belnet.be should be removed from mirrors or a fall-back offered #3784

kwisatz opened this issue Oct 15, 2023 · 13 comments
Assignees
@dduportal
Labels
pkg.jenkins.io
Milestone
infra-team-sync-2...

Comments

@kwisatz
Copy link

kwisatz commented Oct 15, 2023 •
edited

Service(s)

pkg.jenkins.io

Summary

Our CI server, which is hosted in France is made to download from that server and it or the IP range is uses seems to have been blocked by ftp.belnet.be (I can access the site just fine over my local ISP).

See https://community.jenkins.io/t/how-to-hardconfig-the-mirror-to-use/10099 and https://community.jenkins.io/t/issue-while-upgrading-plugins-on-latest-jenkins/9846 for recent reports by other people, as well as a somewhat related https://groups.google.com/g/jenkins-infra/c/C7cW3MKwR0I?pli=1.

The following packages will be upgraded:
  jenkins
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 88.9 MB of archives.
After this operation, 55.3 kB disk space will be freed.
Do you want to continue? [Y/n]
Err:1 https://pkg.jenkins.io/debian binary/ jenkins 2.427
  Could not connect to ftp.belnet.be:443 (193.190.198.27), connection timed out
E: Failed to fetch https://ftp.belnet.be/mirror/jenkins/debian/jenkins_2.427_all.deb  Could not connect to ftp.belnet.be:443 (193.190.198.27), connection timed out
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

vs.

kwisatz@thufir:~$ curl -Is https://ftp.belnet.be/mirror/jenkins/debian/
HTTP/2 200
server: nginx
date: Sun, 15 Oct 2023 14:54:48 GMT
content-type: text/html

I can understand that someone wants to protect their infrastructure from attacks that might have come from a hoster's IP range, but IMHO what they do is too aggressive and it makes belnet.be no longer a valid mirror.

pkg.jenkins.io does not seem to offer any alternatives or fallback options. Meaning I am unable to upgrade jenkins normally (or automatically), which puts me (and everyone with the same issue) at risk.

Reproduction steps

root@fyrine:~# apt update
Hit:2 http://mirrors.online.net/debian bullseye InRelease
Get:3 https://download.docker.com/linux/debian bullseye InRelease [43.3 kB]
Get:4 http://security.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:5 http://deb.debian.org/debian bullseye-backports InRelease [49.0 kB]
Hit:1 https://packages.icinga.com/debian icinga-bullseye InRelease
Get:6 http://httpredir.debian.org/debian bullseye-updates InRelease [44.1 kB]
Ign:7 https://pkg.jenkins.io/debian binary/ InRelease
Hit:8 https://pkg.jenkins.io/debian binary/ Release
Fetched 185 kB in 2s (112 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.
root@fyrine:~# apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  jenkins
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 88.9 MB of archives.
After this operation, 55.3 kB disk space will be freed.
Do you want to continue? [Y/n]
Err:1 https://pkg.jenkins.io/debian binary/ jenkins 2.427
  Could not connect to ftp.belnet.be:443 (193.190.198.27), connection timed out
E: Failed to fetch https://ftp.belnet.be/mirror/jenkins/debian/jenkins_2.427_all.deb  Could not connect to ftp.belnet.be:443 (193.190.198.27), connection timed out
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
@kwisatz kwisatz added the triage Incoming issues that need review label Oct 15, 2023
@dduportal dduportal self-assigned this Oct 16, 2023
@dduportal dduportal added this to the infra-team-sync-2023-10-17 milestone Oct 16, 2023
@dduportal dduportal removed the triage Incoming issues that need review label Oct 16, 2023
@dduportal
Copy link
Contributor

Hi @kwisatz thanks for pointing this out.

As a temporary measure, we've dsiabled ftp.belnet.be and we'll check with them:

$ mirrorbits list
Identifier              STATE           SINCE
tsinghua.edu.cn         up              (Sat, 14 Oct 2023 12:20:16 UTC)
ftp.belnet.be           disabled        (Fri, 06 Oct 2023 06:39:26 UTC)
yamagata-u.ac.jp        up              (Thu, 05 Oct 2023 20:42:23 UTC)
rwth-aachen.de          up              (Fri, 08 Sep 2023 06:07:26 UTC)
ftp-chi.osuosl.org      up              (Sat, 12 Aug 2023 09:03:41 UTC)
ftp-nyc.osuosl.org      up              (Sat, 12 Aug 2023 00:39:32 UTC)
archives.jenkins.io     up              (Fri, 16 Jun 2023 17:26:59 UTC)
servanamanaged.com      up              (Tue, 28 Feb 2023 20:42:14 UTC)
xmission.org            up              (Sun, 12 Feb 2023 09:59:07 UTC)

Can you retry a download?

A nice tip: for any package/plugin download on Jenkins, you can add a custom query string to check the list of mirrors which are able to serve a given file. For instance: https://get.jenkins.io/plugins/ant/497.v94e7d9fffa_b_9/ant.hpi?mirrorlist

pkg.jenkins.io does not seem to offer any alternatives or fallback options.

Godd point. cc @smerle33 @lemeurherve @MarkEWaite for info: this is another example and good reason to update our fallback strategy: I'll take care of documenting and adding archives.jenkins.io and check it.

Meaning I am unable to upgrade jenkins normally (or automatically), which puts me (and everyone with the same issue) at risk.

@kwisatz would your company or you be able to help us by providing (or pointing us) a download mirror in France (free or sponsored)? That would help to provide localized downloads by countries, closer to your infrastructure to leverage the risk of you and your team (as you depend on a freely maintained open source project only living by donation and sponsorships).

@kwisatz
Copy link
Author

kwisatz commented Oct 16, 2023 •
edited

I don't think that it would make a lot of sense for us to supply a mirror in France, even if we happen to run our CI infrastructure over there. There are massive players like OVH or Scaleway that should be in a much better position to do so.

We're just a tiny development company really and I would say, if we'd set up a mirror, then it would make more sense, speaking of scale, not of personal benefits in this case, to do this within the Luxembourg region.
What is the difference between sponsored and free? Can you point me to more info on that?

@dduportal
Copy link
Contributor

What is the difference between sponsored and free? Can you point me to more info on that?

Both cases are "sponsors".

With the wording I used (which might not be perfect English), I meant:

  • "free" is like what Belnet provides: they set up the mirror (an HTTP web server running 24h/24h, with a regular sync from the Jenkins mirror references such as archives.jenkins.io to update their mirror). The Jenkins project "only" has to setup the mirror system to add their HTTP URL (and an rsync or ftp URL to scan their repository) so it's free.
  • "sponsored" is like DigitalOcean: The Jenkins infrastructure team requested a sponsoring and they gave us credits in their cloud. In that case, the Jenkins infra team has to set up and maintain a mirror themselves (archives.jenkins.io). It's not that much effort but requires a bit more planning (but it is also really useful).

I don't think that it would make a lot of sense for us to supply a mirror in France, even if we happen to run our CI infrastructure over there. There are massive players like OVH or Scaleway that should be in a much better position to do so.

No problem, I'm opening the discussion without any goal except reminding us that Jenkins is a project only living from sponsors 🤗

But if by any mean you know an university, a company or an organization whom could provide one of those 2 (or who could be a member of the CDF - https://cd.foundation/members/join/) it would help to achieve your goals (and ours ;) ).

Btw @kwisatz , did the temporary measure unblocked you?

@kwisatz
Copy link
Author

kwisatz commented Oct 16, 2023

Btw @kwisatz , did the temporary measure unblocked you?

@dduportal I had installed the updated .deb manually yesterday, but I downgraded to test this:

The following packages will be upgraded:
  jenkins
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 88.9 MB of archives.
After this operation, 20.5 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 https://pkg.jenkins.io/debian binary/ jenkins 2.427 [88.9 MB]
Fetched 88.9 MB in 8s (11.5 MB/s)
Reading changelogs... Done
(Reading database ... 95899 files and directories currently installed.)
Preparing to unpack .../archives/jenkins_2.427_all.deb ...
Unpacking jenkins (2.427) over (2.426) ...
Setting up jenkins (2.427) ...

So, yes, thank you!

@dduportal
Copy link
Contributor

Hello @kwisatz , could you share with us your public outbound IP(s) please? If you want it to be private, please send to jenkins-infra-team@googlegroups.com (this is the Jenkins infra private list).

We need this to identify your network to check with the Belnet maintainers.

@kwisatz
Copy link
Author

kwisatz commented Oct 20, 2023

@dduportal I've sent an email.

@dduportal
Copy link
Contributor

dduportal commented Oct 20, 2023 •
edited

@dduportal I've sent an email.

thanks! email received, we're going to check 2 things from now:

  • Contacting belnet to check if they have blocked this IP or an associated range
  • Checking the connectivity from the same ISP (not mentioned publicly) to see if we can reproduce the problem on our own

@dduportal
Copy link
Contributor

@kwisatz in parallel, could share with us the results of:

curl --verbose --output /dev/null https://ftp.belnet.be/mirror/jenkins/

and

traceroute ftp.belnet.be

?

@kwisatz
Copy link
Author

kwisatz commented Oct 21, 2023 •
edited 

root@fyrine:~# curl --verbose --output /dev/null https://ftp.belnet.be/mirror/jenkins/ 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 193.190.198.27:443...
*   Trying 2001:6a8:3c80::27:443...
* Immediate connect fail for 2001:6a8:3c80::27: Cannot assign requested address
  0     0    0     0    0     0      0      0 --:--:--  0:02:09 --:--:--     0* connect to 193.190.198.27 port 443 failed: Connection timed out
* Failed to connect to ftp.belnet.be port 443: Connection timed out
  0     0    0     0    0     0      0      0 --:--:--  0:02:09 --:--:--     0
* Closing connection 0
curl: (28) Failed to connect to ftp.belnet.be port 443: Connection timed out

Trace route with the first hop obfuscated:

root@fyrine:~# traceroute ftp.belnet.be
traceroute to ftp.belnet.be (193.190.198.27), 30 hops max, 60 byte packets
 1  ***-***-**-1.rev.poneytelecom.eu (***.***.**.1)  0.358 ms  0.469 ms  0.605 ms
 2  195.154.2.0 (195.154.2.0)  0.326 ms  0.416 ms 195.154.2.2 (195.154.2.2)  0.399 ms
 3  51.158.8.70 (51.158.8.70)  0.425 ms 51.158.8.68 (51.158.8.68)  0.455 ms 51.158.8.74 (51.158.8.74)  0.738 ms
 4  be4752.rcr21.b039311-0.par04.atlas.cogentco.com (149.6.165.65)  0.590 ms  0.629 ms be4751.rcr21.b022890-0.par04.atlas.cogentco.com (149.6.164.41)  0.661 ms
 5  be3750.ccr31.par04.atlas.cogentco.com (154.54.60.201)  1.367 ms be2152.ccr32.par04.atlas.cogentco.com (154.54.61.37)  1.816 ms be3739.ccr31.par04.atlas.cogentco.com (154.54.60.185)  69.638 ms
 6  be2103.ccr42.par01.atlas.cogentco.com (154.54.61.21)  3.482 ms be2102.ccr41.par01.atlas.cogentco.com (154.54.61.17)  1.480 ms be3183.ccr41.par01.atlas.cogentco.com (154.54.38.65)  4.534 ms
 7  be3674.rcr21.bru01.atlas.cogentco.com (130.117.48.234)  6.730 ms  6.850 ms be3675.rcr21.bru01.atlas.cogentco.com (154.54.57.166)  6.715 ms
 8  be2525.nr61.b015929-1.bru01.atlas.cogentco.com (154.25.12.246)  7.357 ms  7.403 ms  7.483 ms
 9  149.11.150.2 (149.11.150.2)  6.796 ms  6.804 ms  6.769 ms
10  * * *
11  * * *

@dduportal
Copy link
Contributor

Sent an email to the FTP belnet maintainers, let's wait for their answer.

@dduportal
Copy link
Contributor

Hi @kwisatz , we receieved an answer from Belnet: your IP is in a range which was blocked due to DDOS from some IPs in it.

They removed the firewall rule: can you confirm it is working for you with a curl https://ftp.belnet.be/mirror/jenkins/debian/jenkins_2.427_all.deb --verbose command please?

@kwisatz
Copy link
Author

kwisatz commented Nov 6, 2023

Looking good!

root@fyrine:~# curl https://ftp.belnet.be/mirror/jenkins/debian/jenkins_2.427_all.deb --verbose
*   Trying 193.190.198.27:443...
* Connected to ftp.belnet.be (193.190.198.27) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=BE; ST=Bruxelles-Capitale, R���gion de; O=R���seau T���l���matique Belge de la Recherche (Belnet); CN=ftp.belnet.be
*  start date: Sep 12 00:00:00 2023 GMT
*  expire date: Sep 11 23:59:59 2024 GMT
*  subjectAltName: host "ftp.belnet.be" matched cert's "ftp.belnet.be"
*  issuer: C=NL; O=GEANT Vereniging; CN=GEANT OV ECC CA 4
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55e455598620)
> GET /mirror/jenkins/debian/jenkins_2.427_all.deb HTTP/2
> Host: ftp.belnet.be
> user-agent: curl/7.74.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< server: nginx
< date: Mon, 06 Nov 2023 10:56:42 GMT
< content-type: application/octet-stream
< content-length: 88906552
< last-modified: Tue, 10 Oct 2023 12:54:54 GMT
< etag: "6525499e-54c9b38"
< accept-ranges: bytes
[…]

@dduportal
Copy link
Contributor

Looking good!

root@fyrine:~# curl https://ftp.belnet.be/mirror/jenkins/debian/jenkins_2.427_all.deb --verbose
*   Trying 193.190.198.27:443...
* Connected to ftp.belnet.be (193.190.198.27) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=BE; ST=Bruxelles-Capitale, R���gion de; O=R���seau T���l���matique Belge de la Recherche (Belnet); CN=ftp.belnet.be
*  start date: Sep 12 00:00:00 2023 GMT
*  expire date: Sep 11 23:59:59 2024 GMT
*  subjectAltName: host "ftp.belnet.be" matched cert's "ftp.belnet.be"
*  issuer: C=NL; O=GEANT Vereniging; CN=GEANT OV ECC CA 4
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55e455598620)
> GET /mirror/jenkins/debian/jenkins_2.427_all.deb HTTP/2
> Host: ftp.belnet.be
> user-agent: curl/7.74.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< server: nginx
< date: Mon, 06 Nov 2023 10:56:42 GMT
< content-type: application/octet-stream
< content-length: 88906552
< last-modified: Tue, 10 Oct 2023 12:54:54 GMT
< etag: "6525499e-54c9b38"
< accept-ranges: bytes
[…]

Thanks for confirmation. We've enabled the mirror again in get.jenkins.io (mirrorbits scan -enable) so it should be serving content.

I'm closing the issue, feel free to reopen if you see a new problem!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dduportal dduportal

Labels
pkg.jenkins.io
Projects
None yet
Milestone
infra-team-sync-2023-11-07
Development

No branches or pull requests
2 participants
@kwisatz @dduportal