Go packages were inaccessible from origin server on Oct 28, 2019 from 3:37 am to 7:56 am (UTC−04) #31
Hi, I wonder if this can be fixed again. I see you already faced this once. I guess our servers are in Japan.
Thanks a lot for letting me know. There was an issue on the web server causing the website to be unavailable. It should be fixed now. Please try again and let me know if you're still seeing any issues.
I'm going to look into improving it to prevent (and detect more quickly) this kind of problem in the future.
I suggest using a module proxy, for example the Go module mirror (https://proxy.golang.org), in order to be able to download modules even when the origin server is temporarily unavailable. @katiehockman's excellent GopherCon 2019 talk covered how module proxies can help mitigate issues such as this one in more detail.
Here's a timeline of the outage, showing a graph of
(Times are in EDT, aka UTC−04 timezone.)
The root cause was a bug in the
The HTTPS proxy program is not setup to be automatically restarted on crash, so all requests stopped being served until it was manually restarted. If automatic restarts were implemented, the outage would've been largely mitigated, but may have been less noticeable and less likely the root cause would be found and fixed (since it'd be easier to ignore). My personal website prioritizes experimentation and development over stability, and so automatic restarts are not used.
I've also added an alert that should help notify me of similar issues in the future.
As mentioned in #31 (comment), if reliability of your build is of high importance, then it's recommended to use a caching module proxy (such as the Go module mirror at https://proxy.golang.org), so that your module's build can be successful even when some origin servers are temporary unavailable. My personal website only has a 95%+ uptime SLA.
Closing since this is resolved.