Caddy反代的ubuntu镜像经常502

[htfy96]

类别：Bug

现象：

59.78.45.192 - - [03/Jun/2018:15:43:05 +0000] "GET /ubuntu/dists/bionic/InRelease HTTP/1.1" 200 242048
59.78.45.192 - - [03/Jun/2018:15:43:05 +0000] "GET /ubuntu/dists/bionic-updates/InRelease HTTP/1.1" 200 83235
59.78.45.192 - - [03/Jun/2018:15:43:05 +0000] "GET /ubuntu/dists/bionic-backports/InRelease HTTP/1.1" 200 74562
59.78.45.192 - - [03/Jun/2018:15:43:05 +0000] "GET /ubuntu/dists/bionic-security/InRelease HTTP/1.1" 200 83237
59.78.45.192 - - [03/Jun/2018:15:43:09 +0000] "GET /ubuntu/dists/bionic/universe/source/by-hash/SHA256/f05cceba96616bd3089a32361c0114bc0ebfd2908f759d5a6e97d8a458e2dc28 HTTP/1.1" 200 11521496
59.78.45.192 - - [03/Jun/2018:15:43:10 +0000] "GET /ubuntu/dists/bionic/main/binary-amd64/by-hash/SHA256/ff7fd80e902a1acfba06e7b513711da31abe915d95b3ba39ce198e02efd209e5 HTTP/1.1" 502 16
59.78.45.192 - - [03/Jun/2018:15:43:10 +0000] "GET /ubuntu/dists/bionic/multiverse/binary-amd64/by-hash/SHA256/910cb989ed0e55b8c98589881a57730a36f074123991366c0263e61582a0e156 HTTP/1.1" 502 16
59.78.45.192 - - [03/Jun/2018:15:43:10 +0000] "GET /ubuntu/dists/bionic/universe/binary-amd64/by-hash/SHA256/ca221e8754c933c636b6c0a344617e3444a7c8cb0982ca97725cda9b7bfe1e6a HTTP/1.1" 502 16
59.78.45.192 - - [03/Jun/2018:15:43:10 +0000] "GET /ubuntu/dists/bionic/restricted/binary-amd64/by-hash/SHA256/81b9542ff39f796dd83159d5ef02161232ae0b766538d8fc02a299fc1ced1f4d HTTP/1.1" 502 16
59.78.45.192 - - [03/Jun/2018:15:43:10 +0000] "GET /ubuntu/dists/bionic-updates/universe/source/by-hash/SHA256/649ad1a2102b4a08f2ec6d3fc153abc8b8afc310e1677160282098b6dabeaf97 HTTP/1.1" 502 16

Caddy配置：

{{ define "reverse_proxy_common_proxy_config" }}
        max_conns 100
        fail_timeout 5s
        max_fails 3
        header_upstream X-Real-IP {remote}                                                                                                                                                                         
        header_upstream X-Forwarded-For {remote}
        header_upstream X-Forwarded-Proto {scheme}
{{ end }}

初步猜测是因为达到了max_fails而爆炸。

但是ftp.sjtu.edu.cn并没有报错，说明网络是正常的，只有可能http会出现问题。联想到Caddy之前的智障行为：把客户端终止链接看作后端fail、把4xx报错认为是后端爆炸，初步断定是后端返回了4xx。但合法请求为什么会返回4xx呢？于是开始抓了几个包：

GET /ubuntu/dists/bionic/InRelease HTTP/1.1
Host: ftp.sjtu.edu.cn
User-Agent: Debian APT-HTTP/1.3 (1.6.1)
Accept: text/*
Cache-Control: max-age=0
Connection: close
If-Range: Thu, 26 Apr 2018 23:38:40 GMT
Range: bytes=242048-
X-Forwarded-For: 59.78.45.192
X-Forwarded-Proto: https
X-Real-Ip: 59.78.45.192

HTTP/1.1 416 Requested Range Not Satisfiable
Server: nginx/1.13.12
Date: Sun, 03 Jun 2018 08:13:54 GMT
Content-Type: text/html
Content-Length: 214
Connection: close
Content-Range: bytes */242048

问题就出自这里。由于apt的一个bug，即使在正常工作的情况下也可能产生大量的416请求，而apt认为这些是合法的。这些416又让caddy认为后端挂了，于是阻塞了整个反代服务。

临时解决办法：关闭后端max_fails