Very high memory commit amount

[nmehlei]

With the newest version (2.8.19.1), we are experiencing very high amounts of committed memory for redis, as apparent in the following screenshot: https://dl.dropboxusercontent.com/u/19676954/RedisPerfMonScreenshot.png

Based on INFO output (used_memory_human), memory usage is around 692 MB, but as can be seen in the screenshot the Commit amount for the redis process is more than 3 GB.

This doesn't look like expected behavior and currently forces us to have massive memory reserves even though our dataset would fit in a fraction of that.

[nmehlei]

I managed to figure out a few additional things. The redis server is configured to persist via snapshot aswell as appendonly, but since the last crash ( issue: #167 ) it did not do either of those, even when explicitly forcing it with bgsave etc.. While panicking because of possible data loss, I configured and started a slave which replicated the whole dataset and managed to save it correctly. Afterwards, possibly because replication forced the saving in another way, the persisting of the master worked again and the memory commit amount went down to a normal amount.

[orangemocha]

Was the AOF file being rewritten while you observed the high memory commit amount?
During the AOF rewrite process, Redis accumulates the new writes in an in-memory buffer: So depending on how long it takes to rewrite the AOF, if your write load is high, the committed memory can grow much larger than the data set. This is part of the original Redis design and not specific to the Windows version.

You should be able to verify the size of the buffer with the INFO command, in the aof_rewrite_buffer_length field.

[nmehlei]

I don't think so, but I am not quite sure. The high memory commit amount was in effect for at least 12 hours and all persisting attempts seem to not even have been started (no log entries whatsoever about persisting), but INFO showed the following:

aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_current_size:5182672645
aof_base_size:1062854500
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0

Which looks like nothing was in progress.

[orangemocha]

@nmehlei how is memory configured in the configuration file? Do you set maxheap, maxmemory? How much physical RAM does the machine have?

[enricogior]

Closing this for inactivity and since it refers to an old release.
Feel free to reopen if needed.
Thank you.