New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Caddy反代的ubuntu镜像经常502 #19

Open
htfy96 opened this Issue Jun 3, 2018 · 0 comments

Comments

Projects
None yet
1 participant
@htfy96
Copy link
Contributor

htfy96 commented Jun 3, 2018

类别:Bug

现象:

59.78.45.192 - - [03/Jun/2018:15:43:05 +0000] "GET /ubuntu/dists/bionic/InRelease HTTP/1.1" 200 242048
59.78.45.192 - - [03/Jun/2018:15:43:05 +0000] "GET /ubuntu/dists/bionic-updates/InRelease HTTP/1.1" 200 83235
59.78.45.192 - - [03/Jun/2018:15:43:05 +0000] "GET /ubuntu/dists/bionic-backports/InRelease HTTP/1.1" 200 74562
59.78.45.192 - - [03/Jun/2018:15:43:05 +0000] "GET /ubuntu/dists/bionic-security/InRelease HTTP/1.1" 200 83237
59.78.45.192 - - [03/Jun/2018:15:43:09 +0000] "GET /ubuntu/dists/bionic/universe/source/by-hash/SHA256/f05cceba96616bd3089a32361c0114bc0ebfd2908f759d5a6e97d8a458e2dc28 HTTP/1.1" 200 11521496
59.78.45.192 - - [03/Jun/2018:15:43:10 +0000] "GET /ubuntu/dists/bionic/main/binary-amd64/by-hash/SHA256/ff7fd80e902a1acfba06e7b513711da31abe915d95b3ba39ce198e02efd209e5 HTTP/1.1" 502 16
59.78.45.192 - - [03/Jun/2018:15:43:10 +0000] "GET /ubuntu/dists/bionic/multiverse/binary-amd64/by-hash/SHA256/910cb989ed0e55b8c98589881a57730a36f074123991366c0263e61582a0e156 HTTP/1.1" 502 16
59.78.45.192 - - [03/Jun/2018:15:43:10 +0000] "GET /ubuntu/dists/bionic/universe/binary-amd64/by-hash/SHA256/ca221e8754c933c636b6c0a344617e3444a7c8cb0982ca97725cda9b7bfe1e6a HTTP/1.1" 502 16
59.78.45.192 - - [03/Jun/2018:15:43:10 +0000] "GET /ubuntu/dists/bionic/restricted/binary-amd64/by-hash/SHA256/81b9542ff39f796dd83159d5ef02161232ae0b766538d8fc02a299fc1ced1f4d HTTP/1.1" 502 16
59.78.45.192 - - [03/Jun/2018:15:43:10 +0000] "GET /ubuntu/dists/bionic-updates/universe/source/by-hash/SHA256/649ad1a2102b4a08f2ec6d3fc153abc8b8afc310e1677160282098b6dabeaf97 HTTP/1.1" 502 16

Caddy配置:

{{ define "reverse_proxy_common_proxy_config" }}
        max_conns 100
        fail_timeout 5s
        max_fails 3
        header_upstream X-Real-IP {remote}                                                                                                                                                                         
        header_upstream X-Forwarded-For {remote}
        header_upstream X-Forwarded-Proto {scheme}
{{ end }}

初步猜测是因为达到了max_fails而爆炸。

但是ftp.sjtu.edu.cn并没有报错,说明网络是正常的,只有可能http会出现问题。联想到Caddy之前的智障行为:把客户端终止链接看作后端fail把4xx报错认为是后端爆炸,初步断定是后端返回了4xx。但合法请求为什么会返回4xx呢?于是开始抓了几个包:

GET /ubuntu/dists/bionic/InRelease HTTP/1.1
Host: ftp.sjtu.edu.cn
User-Agent: Debian APT-HTTP/1.3 (1.6.1)
Accept: text/*
Cache-Control: max-age=0
Connection: close
If-Range: Thu, 26 Apr 2018 23:38:40 GMT
Range: bytes=242048-
X-Forwarded-For: 59.78.45.192
X-Forwarded-Proto: https
X-Real-Ip: 59.78.45.192
HTTP/1.1 416 Requested Range Not Satisfiable
Server: nginx/1.13.12
Date: Sun, 03 Jun 2018 08:13:54 GMT
Content-Type: text/html
Content-Length: 214
Connection: close
Content-Range: bytes */242048

问题就出自这里。由于apt的一个bug,即使在正常工作的情况下也可能产生大量的416请求,而apt认为这些是合法的。这些416又让caddy认为后端挂了,于是阻塞了整个反代服务。

临时解决办法:关闭后端max_fails

@htfy96 htfy96 added the bug label Jun 3, 2018

@htfy96 htfy96 referenced this issue Jun 3, 2018

Closed

Weekly-78 #53

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment