Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zoekt-webserver regularly using 50GB memory, OOMing at 60GB #3792

Closed
slimsag opened this issue May 2, 2019 · 12 comments

Comments

@slimsag
Copy link
Member

commented May 2, 2019

On an xlarge customer deployment (https://app.hubspot.com/contacts/2762526/company/407948923) I am observing zoekt-webserver regularly using ~50GB memory and sometimes OOMing when it goes over 60GB which is the limit we have in place. We've mitigated by bumping up to 80GB but it seems a little crazy to have to do this.

Is there a way we can tell Zoekt "you can use N GB memory, but after that slow down a bit so you don't OOM"? Or is there any low-hanging fruit w.r.t. zoekt-webserver that would reduce memory consumption?

cc @sourcegraph/code-search @sourcegraph/core-services

@slimsag slimsag added the search label May 2, 2019

@slimsag slimsag self-assigned this May 2, 2019

slimsag added a commit to sourcegraph/deploy-sourcegraph-docker that referenced this issue May 2, 2019
@keegancsmith

This comment has been minimized.

Copy link
Member

commented May 3, 2019

Maybe you can try and increase the shard size flag? For a large repository I believe it may result in overall smaller indexes and lower memory usage in a request. Another possibility is reducing the number of results found per shard.

I don't think there is low hanging fruit other than what I mentioned, but maybe an inspection of some memory traces would reveal some. Otherwise there are some likely higher level changes which can be made:

  • Make indexes mmaped as well.
  • Horizontally scale zoekt-webserver.
  • Smarter decisions in which shards to search and in what order.
@slimsag

This comment has been minimized.

Copy link
Member Author

commented May 6, 2019

@keegancsmith I briefly looked at CLI flags for zoekt-webserver and zoekt-indexserver but didn't find a shard flag size. Can you point me to it? Also, what would you recommend trying to set it to? If we alter it how would it affect the existing indexes produced by Zoekt? (would it need to re-index or something)

@keegancsmith

This comment has been minimized.

Copy link
Member

commented May 6, 2019

@keegancsmith I briefly looked at CLI flags for zoekt-webserver and zoekt-indexserver but didn't find a shard flag size. Can you point me to it?

-shard_limit https://sourcegraph.com/github.com/sourcegraph/zoekt/-/blob/cmd/flags.go#L42:26
It would be for the index server. You would probably have to update our indexserver to set the field, we justrely on the default. @ijsnow recently added functionality allowing us to pass flags from sourcegraph down to the indexserver, so that may be the best place to do it.

Also, what would you recommend trying to set it to?

Zoekt divides its parallelism up by shards (ie there is one goroutine per shard). So how long it takes to search a shard is the critical path for how quick zoekt is assuming you had infinite cores. 100mb is quite small. As an educated guess I would suspect 1G would still result in fast searches.

If we alter it how would it affect the existing indexes produced by Zoekt? (would it need to re-index or something)

I believe all indexes will be recomputed (assuming @ijsnow work on reindexing based on command line flags has gone in). If it hasn't, it will only re-index when there is a new commit to index.

@slimsag slimsag added this to the 3.7 milestone Jul 9, 2019

@slimsag

This comment has been minimized.

Copy link
Member Author

commented Jul 9, 2019

Note ongoing work here: google/zoekt#86

@slimsag

This comment has been minimized.

Copy link
Member Author

commented Aug 14, 2019

@sourcegraph/core-services I am reassigning to you for tracking purposes. I know you're already doing some work on this in google/zoekt#86

It sounds like a good first step here would be what @tsenart suggested in google/zoekt#86 (comment) with just setting GOGC=50 in our zoekt-webserver and zoekt-indexserver containers? Any reason to not do that?

@slimsag

This comment has been minimized.

Copy link
Member Author

commented Aug 14, 2019

setting GOGC=50 in our zoekt-webserver and zoekt-indexserver containers

would be good to land this in 3.7 if so

@tsenart

This comment has been minimized.

Copy link
Contributor

commented Aug 14, 2019

would be good to land this in 3.7 if so

That sounds like a good idea. Can you land those patches?

@ijt

This comment has been minimized.

Copy link
Contributor

commented Aug 14, 2019

@slimsag

This comment has been minimized.

Copy link
Member Author

commented Aug 14, 2019

(I am doing so)

@ijt

This comment has been minimized.

Copy link
Contributor

commented Aug 14, 2019

slimsag added a commit to sourcegraph/deploy-sourcegraph-docker that referenced this issue Aug 14, 2019
slimsag added a commit to sourcegraph/deploy-sourcegraph-docker that referenced this issue Aug 14, 2019
slimsag added a commit to sourcegraph/deploy-sourcegraph-docker that referenced this issue Aug 14, 2019
update zoekt images (#36)
* update zoekt-indexserver image

Setting GOGC=50, see sourcegraph/sourcegraph#3792

* update zoekt-webserver image

Setting GOGC=50, see sourcegraph/sourcegraph#3792
slimsag added a commit that referenced this issue Aug 14, 2019
@slimsag

This comment has been minimized.

Copy link
Member Author

commented Aug 14, 2019

@slimsag

This comment has been minimized.

Copy link
Member Author

commented Aug 14, 2019

We'll see how big of an impact this has and, if insufficient, will re-open this.

@slimsag slimsag closed this Aug 14, 2019

slimsag added a commit that referenced this issue Aug 14, 2019
Set GOGC=50 on Zoekt (#5199)
* cmd/server/shared: set GOGC=50 on Zoekt (see #3792)

* dev: set GOGC=50 on Zoekt (see #3792)

* CHANGELOG: mention Zoekt GOGC=50 memory consumption change

* fixup
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.