301 Redirect Loop due to unknown CDN nodes

[baldoarturo]

Hello everybody, hope you are doing great
This is Arturo Baldo on behalf of AS 262187.
I am experiencing a loop of 301 redirects since 2 weeks ago, i spoke to stackpath/highwinds cdn and they say it could be a misconfiguration at jQuery's end
Is there anything we can do to help in order to fix it?

[brianwarner]

Hi Arturo, we've been working on this, it appears there are some new CDN nodes that came online. Our upstream server was unaware of them, which would likely cause the 301. We've updated the configuration and have purged the CDN, and hopefully this will resolve the issues once various caches clear.

[Krinkle]

From @marcosnils at #52 (comment)

* Connected to code.jquery.com (2001:4de0:ac19::1:b:2a) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1 […]
* ALPN, server accepted to use h2 […]
> HEAD /jquery-3.4.1.min.js HTTP/2
> Host: code.jquery.com
> user-agent: curl/7.68.0 […]
< HTTP/2 301  […]
< location: https://code.jquery.com/jquery-3.4.1.min.js
< cache-control: max-age=2592000
< cache-control: public
< access-control-allow-origin: *
< x-hw: 1608079376.dop210.ez1.t,1608079376.cds205.ez1.hn,1608079376.cds221.ez1.c […]

$ dig code.jquery.com
;; ANSWER SECTION:
code.jquery.com.	181	IN	CNAME	cds.s5x3j6q5.hwcdn.net.
cds.s5x3j6q5.hwcdn.net.	183	IN	A	209.197.3.24

$ dig AAAA code.jquery.com
;; ANSWER SECTION:
code.jquery.com.	167	IN	CNAME	cds.s5x3j6q5.hwcdn.net.
cds.s5x3j6q5.hwcdn.net.	167	IN	AAAA	2001:4de0:ac19::1:b:2a
cds.s5x3j6q5.hwcdn.net.	167	IN	AAAA	2001:4de0:ac19::1:b:1a
cds.s5x3j6q5.hwcdn.net.	167	IN	AAAA	2001:4de0:ac19::1:b:3b
cds.s5x3j6q5.hwcdn.net.	167	IN	AAAA	2001:4de0:ac19::1:b:3a
cds.s5x3j6q5.hwcdn.net.	167	IN	AAAA	2001:4de0:ac19::1:b:1b
cds.s5x3j6q5.hwcdn.net.	167	IN	AAAA	2001:4de0:ac19::1:b:2b

[kkatpcc]

This perplexing issue hit us today as well. Trying to connect to anything at https://code.jquery.com via cds.s5x3j6q5.hwcdn.net (209.197.3.24) from our network is presenting us with a 301 redirect, which is causing browsers to give up loading jQuery related assets after ten or so 301 redirect loop attempts.

Any ETA on when the "caches will clear"?

traceroute:

traceroute code.jquery.com
traceroute to cds.s5x3j6q5.hwcdn.net (209.197.3.24), 64 hops max, 52 byte packets
 1  (REDACTED, internal IP)
 2  (REDACTED, internal IP)
 3  74-203-59-13.static.ctl.one (74.203.59.13)  22.065 ms  21.817 ms  22.382 ms
 4  ae14-200g.ar6.sgo1.gblx.net (67.17.99.130)  273.381 ms  279.951 ms  205.278 ms
 5  4.68.75.205 (4.68.75.205)  204.210 ms  234.735 ms  205.456 ms
 6  8.243.188.54 (8.243.188.54)  204.312 ms  282.945 ms  183.126 ms
 7  * * *
 8  vip0x018.map2.ssl.hwcdn.net (209.197.3.24)  189.243 ms  183.086 ms  207.474 ms 

cURL:

curl -vvv https://code.jquery.com
*   Trying 209.197.3.24:443...
* Connected to code.jquery.com (209.197.3.24) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=jquery.org
*  start date: Oct  6 00:00:00 2020 GMT
*  expire date: Oct 16 23:59:59 2021 GMT
*  subjectAltName: host "code.jquery.com" matched cert's "code.jquery.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7ffd75009800)
> GET / HTTP/2
> Host: code.jquery.com
> user-agent: curl/7.72.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 301 
< date: Wed, 16 Dec 2020 23:16:25 GMT
< content-length: 178
< content-type: text/html
< accept-ranges: bytes
< server: nginx
< location: https://code.jquery.com/
< cache-control: max-age=2592000
< cache-control: public
< access-control-allow-origin: *
< x-hw: 1608160585.dop208.sc1.t,1608160585.cds204.sc1.hn,1608160585.cds208.sc1.c
< 
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host code.jquery.com left intact

[mgol]

@kkatpcc Can you check now? If it still fails, can you check on a different browser and after clearing browser cache? This is just so that if it fails, we know whether the fix worked at all and we just need to handle caches now or if it's still broken as it was.

[baldoarturo]

Arturo@ARTURO-NB C:\Users\Arturo
# tracert code.jquery.com

Traza a la dirección cds.s5x3j6q5.hwcdn.net [209.197.3.24]
sobre un máximo de 30 saltos:

  1     2 ms     1 ms    <1 ms  192.168.10.1
  2     5 ms    11 ms     4 ms  192.168.15.1
  3     6 ms     9 ms     3 ms  65-211-80-190.patagoniagreen.com [190.211.80.65]
  4     5 ms     6 ms     6 ms  192.168.80.1
  5     4 ms     4 ms    10 ms  192.168.2.245
  6     *      827 ms    97 ms  200-32-126-37.static.impsat.net.ar [200.32.126.37]
  7     *        *        *     Tiempo de espera agotado para esta solicitud.
  8    21 ms    27 ms    23 ms  4.68.37.33
  9    22 ms    22 ms    26 ms  8.243.137.62
 10     *        *        *     Tiempo de espera agotado para esta solicitud.
 11    20 ms    19 ms    23 ms  vip0x018.map2.ssl.hwcdn.net [209.197.3.24]

Arturo@ARTURO-NB C:\Users\Arturo
# curl -vvv https://code.jquery.com/jquery-3.5.1.slim.min.js
*   Trying 209.197.3.24...
* TCP_NODELAY set
* Connected to code.jquery.com (209.197.3.24) port 443 (#0)
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 186 bytes...
* schannel: sent initial handshake data: sent 186 bytes
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 2/3)
* schannel: encrypted data got 4096
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: encrypted data length: 4026
* schannel: encrypted data buffer: offset 4026 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 5050 length 5050
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 6074 length 6074
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 2/3)
* schannel: encrypted data got 861
* schannel: encrypted data buffer: offset 6935 length 7098
* schannel: sending next handshake data: sending 126 bytes...
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 2/3)
* schannel: encrypted data got 226
* schannel: encrypted data buffer: offset 226 length 7098
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 3/3)
* schannel: stored credential handle in session cache
> GET /jquery-3.5.1.slim.min.js HTTP/1.1
> Host: code.jquery.com
> User-Agent: curl/7.55.1
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 666
* schannel: encrypted data buffer: offset 666 length 103424
* schannel: decrypted data length: 430
* schannel: decrypted data added: 430
* schannel: decrypted data cached: offset 430 length 102400
* schannel: encrypted data length: 207
* schannel: encrypted data cached: offset 207 length 103424
* schannel: decrypted data length: 178
* schannel: decrypted data added: 178
* schannel: decrypted data cached: offset 608 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 608 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 608
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 301 Moved Permanently
< Date: Thu, 17 Dec 2020 12:36:45 GMT
< Connection: Keep-Alive
< Content-Length: 178
< Content-Type: text/html
< Accept-Ranges: bytes
< Server: nginx
< Location: https://code.jquery.com/jquery-3.5.1.slim.min.js
< Cache-Control: max-age=2592000
< Cache-Control: public
< Access-Control-Allow-Origin: *
< X-HW: 1608208604.dop211.ez1.t,1608208604.cds213.ez1.shn,1608208605.dop211.ez1.t,1608208605.cds218.ez1.c
<
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host code.jquery.com left intact

[marcosnils]

Still 301 here.

[mgol]

Thanks, we'll have another look.

[kkatpcc]

@mgol Still a 301 redirect loop here as well, in multiple browsers and cURL.

[kkatpcc]

@mgol, I poked around a bit more, and found that while the default/sole A (IPv4) DNS entry for code.jquery.com -- 209.197.3.24, a.k.a. vip0x018.map2.ssl.hwcdn.net -- is giving us troubles, rigging a cURL call to connect to map2.ssl.hwcdn.net (205.185.208.154 -- without any vip* subdomain/node) works.

Here is the "X-HW" line from running curl -vvv -H 'Host: code.jquery.com' https://map2.ssl.hwcdn.net:

X-HW: 1608229349.dop211.se2.t,1608229349.cds030.se2.shn,1608229349.dop211.se2.t,1608229349.cds020.se2.c

Relevant traceroute, hops after 7 are ignoring ICMP:

traceroute map2.ssl.hwcdn.net
traceroute to 205.185.208.154 (205.185.208.154), 64 hops max, 52 byte packets
 1  (REDACTED, internal IP)  24.145 ms  22.787 ms  25.083 ms
 2  (REDACTED, internal IP)  33.776 ms  24.255 ms  26.103 ms
 3  74-203-59-13.static.ctl.one (74.203.59.13)  21.732 ms  26.765 ms  32.312 ms
 4  ae10-80g.ar8.sea1.gblx.net (207.218.1.246)  27.933 ms  57.589 ms  26.809 ms
 5  * * 4.68.75.30 (4.68.75.30)  31.080 ms
 6  ae-2-3615.edge6.seattle1.level3.net (4.69.219.210)  32.199 ms  62.114 ms  28.782 ms
 7  be3094.ccr22.sea02.atlas.cogentco.com (154.54.10.245)  37.928 ms  37.697 ms  27.791 ms

Hope this helps with troubleshooting.

[kkatpcc]

According to the two-character codes at https://status.stackpath.com/ (StackPath CDN, formerly Highwinds) coupled with the "X-HW" response header (assuming they are accurate) it looks like we are being routed from Portland, Oregon to Santiago, Chile (SC) when trying to connect to code.jquery.com.

While that is not our geographically "closest" route, it may have the lowest latency or number of hops. According to https://www.stackpath.com/why-stackpath/network/ our geographically closest route would be Seattle (SE) and in fact that is what we get when connecting to map2.ssl.hwcdn.net as shown in my previous comment.

My first traceroute comment shows sgo1.gblx.net, which seems to correspond with Santiago, and my second one shows sea1.gblx.net, which seems to correspond with Seattle. @baldoarturo seems to be having issues with the CDN in Buenos Aires, Argentina. Both Santiago, Chile and Buenos Aires, Argentina are in South America, not too terribly far from each other, and their respective CDN endpoints are having issues with 301 redirect loops. Something for StackPath to focus on.

[baldoarturo]

That's correct @kkatpcc, I have been in touch with StackPath / HighWinds NOC and they are blaming this on jQuery's end, so I am not sure how to help. So yeah here we are on github discussing a probable networking issue. I just want to help on this but not sure how.

[kkatpcc]

@baldoarturo yep, no fun this one.

I did notice that along both of our traceroutes to code.jquery.com, both Level3 (4.68.75.205 & 4.68.37.33) and CenturyLink (8.243.188.54 & 8.243.137.62) are involved. Perhaps one of them is either being really slow or completely failed to pick up on the CDN purge?

[mgol]

Let me share more details on what we think is happening. Our current infrastructure for https://code.jquery.com was set up long time ago in a way that we keep a list of IPs belonging to the CDN provider. When a regular user tries to access any page on https://code.jquery.com, the request goes to the CDN provider which then serves the asset if it has it cached. If it does not, the CDN provider first requests that asset directly via https://codeorigin.jquery.com. For all regular users, that page just redirects to https://code.jquery.com to avoid people loading assets from our infrastructure directly (we would not be able to handle the load ourselves). If the CDN provider uses a new IP that we don't recognize, its requests to https://codeorigin.jquery.com are redirected back to ttps://code.jquery.com, triggering a redirect loop.

Now, Highwinds, our current CDN provider, added more IPs to its servers; those IPs were not on that list. We updated the list based on what they document but apparently that page is outdated. We requested the current list of IPs and they provided us a longer list which we applied. Unfortunately, it seems even that list may not be fully up to date as the problem persists.

What we're trying now is to specify a special header that will be included in requests from Highwinds and skip the redirect based on the presence of that header, getting rid of the IP-based approach completely. We hope this will get rid of this issue once & for all.

[Bruno02468]

I'd like to add that this issue is also affecting users from São Paulo (Brazil). My father and I noticed that some sites stopped working, and after some investigation, I ended up here. Same 301 loop.

However, it behaves differently across ISPs and link types:

• My home network (fails)
• My friend's home network (works)
• My VPS (works)

I hope this can be of use. I'll see if I can conduct more tests later and keep this up to date.

[kkatpcc]

@mgol Our previously failing code.jquery.com route to the Santiago CDN seems to be working now, no more 301 redirect loop.

[marcosnils]

Same here. Seems to be working now Em qui, 17 de dez de 2020 22:27, kkatpcc <notifications@github.com> escreveu:

…

@mgol <https://github.com/mgol> Our previously failing code.jquery.com route to the Santiago CDN seems to be working now, no more 301 redirect loop. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#67 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMBLWXWLMHES6HINNBSRYDSVKVX7ANCNFSM4U3E5UDQ> .

[kkatpcc]

Looks like ours was a partial success. While the homepage https://code.jquery.com works, any assets (e.g. https://code.jquery.com/jquery-1.11.3.js) seem to be still stuck in a 301 redirect loop.

[muylomas]

I'm trying to reach any assets (e.g. https://code.jquery.com/jquery-1.11.1.min.js) from Uruguay, South America. Any browser is giving the next problem:

This page isn’t working
code.jquery.com redirected you too many times.
Try clearing your cookies.
ERR_TOO_MANY_REDIRECTS

👉 👉 Today, the homepage https://code.jquery.com works, thing that was not happening yesterday!!

[marcosnils]

Same here. Seems to be working now

👋 just came back to say I didn't check correctly and indeed seems not to be fixed yet. As @kkatpcc says, seems like code.jquery.com works but bundles indeed can't be referenced and still return 301

[kkatpcc]

Update. Some of the assets have started working for us (e.g. https://code.jquery.com/jquery-1.12.4.js) while others (e.g. https://code.jquery.com/ui/1.12.0/jquery-ui.js) are still exhibiting a 301 redirect loop. Moving in the right direction.

[Krinkle]

@muylomas @kkatpcc Thanks, I assume these are cache hits from your nearest CDN nodes. For the past two hours, I've been slowly issuing reload instructions for the entire catalog toward the CDN, which has now completed. Does it seem fine now?

[marcosnils]

👋 just checked some of the URL's that I have and all of them seem to be working

…

On Fri, Dec 18, 2020 at 3:29 PM Timo Tijhof ***@***.***> wrote: @muylomas <https://github.com/muylomas> @kkatpcc <https://github.com/kkatpcc> Thanks, I assume these are cache hits from your nearest CDN nodes. For the past two hours, I've been slowly issuing reload instructions for the entire catalog toward the CDN, which has now completed. Does it seem fine now? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#67 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMBLWT22C4DR32JZNVJLVDSVONP5ANCNFSM4U3E5UDQ> .

[kkatpcc]

@Krinkle It is a little better. The previously tested https://code.jquery.com/ui/1.12.0/jquery-ui.js link is now fine, but https://code.jquery.com/ui/1.12.0/themes/base/jquery-ui.css still does a 301 redirect.

[kkatpcc]

@Krinkle https://code.jquery.com/ui/1.12.0/themes/base/jquery-ui.css is now working. Seems like the overall problem is slowly but surely resolving.

[Krinkle]

Ah, that one isn't linked from https://code.jquery.com/ui/. I'm proactively purging the rest now as well.

[baldoarturo]

Seems to be working from AS262167 and AS3549
Thanks a lot jQuery team!

[muylomas]

Hi, this url https://code.jquery.com/jquery-latest.min.js is not working

[Krinkle]

Hi, this url https://code.jquery.com/jquery-latest.min.js is not working

Tracked at #68 and now resolved. Thanks!

[Krinkle]

@santiagobasulto wrote:

I'm having the same issue from Argentina (100 miles away from Uruguay). The issue is that some versions of JQuery Core return 301 with location to the same URL. This does NOT happen while VPNing through, for example, the US. Seems to be a local (South America thing).

Examples:

$ curl -I https://code.jquery.com/jquery-1.11.0.min.js
HTTP/2 301
location: https://code.jquery.com/jquery-1.11.0.min.js. # This causes the infinite loop

With another version works ok:

$ curl -I https://code.jquery.com/jquery-1.11.3.min.js. # this version works
HTTP/2 200

Could you run these commands to help us understand which CDN node you are being routed through?

dig code.jquery.com

dig AAAA code.jquery.com

curl -vI https://code.jquery.com/jquery-1.11.0.min.js

[kkatpcc]

@Krinkle Found a new straggler that is triggering a 301 redirect loop against a SC node: https://code.jquery.com/jquery-1.9.1.min.js