Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync Gateway memory usage climbs continously #1264

Open
Ariemeth opened this issue Nov 4, 2015 · 10 comments

Comments

@Ariemeth
Copy link

commented Nov 4, 2015

Running the sync gateway built from master(6162dcf) with either go 1.4 or 1.5 running on windows server 2012 with Couchbase community edition server build-4051 running in a 3 node cluster
, continues to increase in memory usage as documents are added at a steady rate of 120 to 150 documents per second.
Documents Memory
103,600 1,032 MB
152,440 1,218 MB
407,000 2,572 MB

The current trend line for the memory growth we are seeing is y=0.0051x + 537.36 where x is number of documents and y is memory consumption in MB.

@adamcfraser

This comment has been minimized.

Copy link
Contributor

commented Nov 4, 2015

What's the channel distribution for those documents? Recent documents are cached in memory per channel - if the total number of channels is rising with the document count, that might be a factor in the memory usage you're seeing.

@Ariemeth

This comment has been minimized.

Copy link
Author

commented Nov 4, 2015

Right now, each document is generating it's own channel. If new channels are cached in memory that would explain why I am seeing this behavior. However the problem I have seen with this, is once the total memory usage maxes out the system RAM, it causes stability issues. Is there a way to set a max for how much should be cached?

@adamcfraser

This comment has been minimized.

Copy link
Contributor

commented Nov 4, 2015

There's a limit on the number of documents being cached per channel (500), but there isn't currently a way to limit the number of channels being cached. This is an area that's getting attention in the upcoming SG release (1.2), but there isn't currently a configuration option to alleviate memory pressure when allocating a very large number of channels.

Let me know if you expect this to be your real-world use case (one channel per doc), and we can discuss options.

@Ariemeth

This comment has been minimized.

Copy link
Author

commented Nov 5, 2015

The current channel for each document is a temporary solution put in place until couchbase-lite.net can filter on documents.

@elevine

This comment has been minimized.

Copy link

commented Nov 5, 2015

We are using this as a workaround until issue #545 is resolved.

For our application, if a user accumulates a lot of deletions, then their initial sync can take a long time (minutes). Our solution is to first do a one time pull replication of the non-deleted documents, then start the continuous replication after that point. The only way I see to accomplish that right now is by having each document get published to a channel using its Id as the channel name, and then the one time pull replicator is given a list of those channels.

For more context on why we chose to do this, see: https://gitter.im/couchbase/mobile?at=55f586634624296d78af04c5

@zgramana zgramana modified the milestones: 1.2.0, 1.3 Nov 6, 2015

@zgramana zgramana modified the milestones: 1.2.0, 1.3 Dec 7, 2015

@ravijk

This comment has been minimized.

Copy link

commented Jan 12, 2016

@adamcfraser limiting number of channels being cached in memory is a real world scenario that would help us a great deal. We have a large number of customers and we are looking to store their booking statements on device so that they can access them offline from local database. Generating individual channel for customer will help us in making sure each local database only has data related to individual customer, however this also means as our customer base grows the number of channels with grow, in some cases quite quickly. Is there a timeline when this feature will be released?

@zgramana zgramana modified the milestones: 1.2.0, 1.3 Feb 17, 2016

@adamcfraser adamcfraser modified the milestones: 1.3, Future May 26, 2016

@kcroaker

This comment has been minimized.

Copy link

commented Aug 4, 2016

Now that 1.3 has been released and doesn't list this bug as a known issue, that that mean this issue is resolved?

@adamcfraser

This comment has been minimized.

Copy link
Contributor

commented Aug 4, 2016

@kcroaker No, this enhancement is not included in 1.3.

@polfer

This comment has been minimized.

Copy link

commented Aug 4, 2016

We're also seeing the slow growth issue on gateway, and also in a situation where we have slow growth in the channel count (not channel per document, but for us, channel for each of one of our specific document types). So I'd throw in an upvote for this as well.

Having said that, under what circumstances will channels get put into the cache following a restart?

@adamcfraser

This comment has been minimized.

Copy link
Contributor

commented Aug 4, 2016

Channels will get put into the cache following a restart if either:

  • A user with access to the channel issues a _changes request (i.e. replicates)
  • A document in that channel is written to the database

@djpongh djpongh removed the docs label Oct 17, 2016

@djpongh djpongh added the performance label May 22, 2017

@djpongh djpongh removed this from the Future milestone Aug 9, 2018

@djpongh djpongh added this to the Iridium milestone Aug 9, 2018

@djpongh djpongh added the P1: high label Aug 30, 2018

@djpongh djpongh modified the milestones: Iridium, Cobalt Nov 19, 2018

@djpongh djpongh modified the milestones: Cobalt, Mercury Dec 13, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.