Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/pkgsite: dealing with excessive load on the worker #41452

Closed
jba opened this issue Sep 17, 2020 · 6 comments
Closed

x/pkgsite: dealing with excessive load on the worker #41452

jba opened this issue Sep 17, 2020 · 6 comments

Comments

@jba
Copy link
Contributor

@jba jba commented Sep 17, 2020

When dealing with many large modules at once, the worker becomes overloaded and OOMs.

Reducing the max concurrency of the queue and/or increasing the number of worker instances can help, but that requires tuning, and often the characteristics change. For example, workers can handle much higher rates when processing many small modules.

Investigate load-shedding and other techniques for making the worker more resilient.

@jba jba added this to the pkgsite/unplanned milestone Sep 17, 2020
@jba jba self-assigned this Sep 17, 2020
@jba
Copy link
Contributor Author

@jba jba commented Sep 17, 2020

We're going to start by tracking the total size of all zip files that are in flight, as a signal for the amount of memory the worker will eventually need to process them. It makes sense to track this rather than actual memory usage because:

  • we can obtain it early in the request path;
  • it leads memory allocation, rather than lagging as would a direct measurement of heap size;
  • it isn't confounded by GC effects.

/cc @findleyr

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 17, 2020

Change https://golang.org/cl/255738 mentions this issue: internal/fetch: refactor to get info separately

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 17, 2020

Change https://golang.org/cl/255737 mentions this issue: internal/proxy: implement GetZipSize

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 17, 2020

Change https://golang.org/cl/255757 mentions this issue: internal/stdlib: Zip takes resolved versionn

gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 17, 2020
Provide an efficient way to get the size of a module zip file
without downloading it.

For golang/go#41452

Change-Id: I9a996efb83146109a31e73b302af1b49f197b43f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/255737
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Sep 17, 2020

Change https://golang.org/cl/255759 mentions this issue: internal/worker: implement load-shedding

gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 18, 2020
Reorganize FetchModule into two functions.

First, one calls GetModuleInfo to get some prelimary,
cheap information, including the size of the zip.

Then one calls FetchModule with the result of GetModuleInfo.

This allows us to make decisions based on the zip size
before downloading the entire zip.

For golang/go#41452

Change-Id: Ifbffa5882488e31a72e78670caf69c3d97a80546
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/255738
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 18, 2020
In the only call to this function, we already know the resolved
version.

For golang/go#41452

Change-Id: I96e5ebdf5acb0aeec45288b25b555e5c6cbb52ee
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/255757
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 18, 2020
Don't fetch a module if the total zip size currently being fetched
exceeds a threshold.

For golang/go#41452

Change-Id: I9f3f12c1daf036754877b78b1eb3607404dc676b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/255759
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
@gopherbot gopherbot added the go.dev label Sep 18, 2020
@julieqiu julieqiu removed the go.dev label Sep 19, 2020
@gopherbot gopherbot added the go.dev label Sep 19, 2020
@julieqiu julieqiu removed the go.dev label Sep 19, 2020
@jba
Copy link
Contributor Author

@jba jba commented Sep 23, 2020

Load-shedding is implemented and seems successful.

@jba jba closed this Sep 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants