Athens leaks etcd leases #1886
Comments
Athens creates a new session for each 'Stash' request and then never closes it. The sessions (etcd leases) are consequently held forever and eventually leads to resource exhaustion on the etcd cluster. Fixes: gomods#1886
Athens creates a new session for each 'Stash' request and then never closes it. The sessions (etcd leases) are consequently held forever and eventually leads to resource exhaustion on the etcd cluster. This has been fixed by closing the acquired session at the end of the request. Fixes: gomods#1886
Athens creates a new session for each 'Stash' request and then never closes it. The sessions (etcd leases) are consequently held forever and eventually leads to resource exhaustion on the etcd cluster. This has been fixed by closing the acquired session at the end of the request. Fixes: gomods#1886
Athens creates a new session for each 'Stash' request and then never closes it. The sessions (etcd leases) are consequently held forever and eventually leads to resource exhaustion on the etcd cluster. This has been fixed by closing the acquired session at the end of the request. Fixes: gomods#1886
Athens creates a new session for each 'Stash' request and then never closes it. The sessions (etcd leases) are consequently held forever and eventually leads to resource exhaustion on the etcd cluster. This has been fixed by closing the acquired session at the end of the request. Fixes: gomods#1886
Athens creates a new session for each 'Stash' request and then never closes it. The sessions (etcd leases) are consequently held forever and eventually leads to resource exhaustion on the etcd cluster. This has been fixed by closing the acquired session at the end of the request. Fixes: gomods#1886
Athens creates a new session for each 'Stash' request and then never closes it. The sessions (etcd leases) are consequently held forever and eventually leads to resource exhaustion on the etcd cluster. This has been fixed by closing the acquired session at the end of the request. Fixes: gomods#1886
Athens creates a new session for each 'Stash' request and then never closes it. The sessions (etcd leases) are consequently held forever and eventually leads to resource exhaustion on the etcd cluster. This has been fixed by closing the acquired session at the end of the request. Fixes: gomods#1886
Athens creates a new session for each 'Stash' request and then never closes it. The sessions (etcd leases) are consequently held forever and eventually leads to resource exhaustion on the etcd cluster. This has been fixed by closing the acquired session at the end of the request. Fixes: gomods#1886
Athens creates a new session for each 'Stash' request and then never closes it. The sessions (etcd leases) are consequently held forever and eventually leads to resource exhaustion on the etcd cluster. This has been fixed by closing the acquired session at the end of the request. Fixes: gomods#1886
Athens creates a new session for each 'Stash' request and then never closes it. The sessions (etcd leases) are consequently held forever and eventually leads to resource exhaustion on the etcd cluster. This has been fixed by closing the acquired session at the end of the request. Fixes: gomods#1886
|
Hello, @uhthomas wanted to know how did you solve this problem ? |
|
@santiagoPinzonD You will have to build Athens from source. FROM golang:1.21.3-alpine AS build
RUN apk update && apk add git
RUN go install github.com/gomods/athens/cmd/proxy@main
FROM gomods/athens:v0.12.1
COPY --from=build /go/bin/proxy /bin/athens-proxy |
|
Please use canary tag which has more fixes and built from master's latest. I will see if I can do a release in coming weeks |
|
Thanks @manugupt1, wasn't aware of this image - though it is a shame that there aren't tags for each commit. Current latest digest for canary: |
|
We do have tags for every commit but under a separate registry: https://hub.docker.com/r/gomods/athens-dev/tags. Feel free to use it if you wanna pin to a specific commit until the next stable version is released. |
|
Done, thanks guys |
|
What specifically are you asking, sorry? There is no database required for Athens. Just use an S3-compatible storage system and the existing etcd cluster. You shouldn't need a separate instance of Athens per GitHub organisation either. |
Describe the bug
Athens creates a new session (lease) for each stash request, and then never closes the session. The leases are then held indefinitely, or until the process is restarted, which causes excessive load on the etcd cluster and eventually overloads it.
The spikes are from restarting Athens.
Error Message
N/A
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Athens should either create a single session for all locks, or close the session. I am not sure which is correct as I could not find great documentation on what is recommended by the etcd authors. I imagine it's probably best just to make sure the session is closed.
Environment (please complete the following information):
Additional context
The text was updated successfully, but these errors were encountered: