Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

VM can't suppert Lists Or sets have much numbers。 #1

Closed
tongxiaojun opened this Issue · 6 comments

2 participants

@tongxiaojun

VM can't suppert Lists Or sets have much numbers。

@antirez
Owner

Hello,

can you please elaborate a bit? Like how much numbers, how to replicate, and the error reported by Redis. Thanks!

@tongxiaojun

when I use command like redis-cli -r 10000000 lpush mylist aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa[1k content]。the redis will use a lot of memory。

@tongxiaojun

redis_version:1.3.3
arch_bits:64
multiplexing_api:epoll
process_id:26560
uptime_in_seconds:106
uptime_in_days:0
connected_clients:2
connected_slaves:0
blocked_clients:0
used_memory:179048675
used_memory_human:170.75M
changes_since_last_save:559778
bgsave_in_progress:0
last_save_time:1264264099
bgrewriteaof_in_progress:0
total_connections_received:39
total_commands_processed:1181478
vm_enabled:1
role:master
vm_conf_max_memory:0
vm_conf_page_size:256
vm_conf_pages:104857600
vm_stats_used_pages:0
vm_stats_swapped_objects:0
vm_stats_swappin_count:1
vm_stats_swappout_count:1
vm_stats_io_newjobs_len:0
vm_stats_io_processing_len:0
vm_stats_io_processed_len:0
vm_stats_io_waiting_clients:0
vm_stats_io_active_threads:0
db0:keys=1,expires=0

this info show redis can't VM one key。

@tongxiaojun

my redis config is vm-max-memory 0
I want use reids list pop push make a simple queue 。
but I find redis can't VM a key lists to many numbers。like 1000 0000 numbers and use a little memory。and it will use nearby 80% system memory at most。and when i run this long time 。
the linux will crash [my max system memory is 4G ]。

@antirez
Owner

Hello, thanks for the info, could you please enter IRC irc.freenode.org #redis for a chat?

@antirez
Owner

Can't remember the details, but this issue is now closed.

@shuge shuge referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@antirez antirez referenced this issue from a commit
@antirez Draft #1 of a new expired keys collection algorithm.
The main idea here is that when we are no longer to expire keys at the
rate the are created, we can't block more in the normal expire cycle as
this would result in too big latency spikes.

For this reason the commit introduces a "fast" expire cycle that does
not run for more than 1 millisecond but is called in the beforeSleep()
hook of the event loop, so much more often, and with a frequency bound
to the frequency of executed commnads.

The fast expire cycle is only called when the standard expiration
algorithm runs out of time, that is, consumed more than
REDIS_EXPIRELOOKUPS_TIME_PERC of CPU in a given cycle without being able
to take the number of already expired keys that are yet not collected
to a number smaller than 25% of the number of keys.

You can test this commit with different loads, but a simple way is to
use the following:

Extreme load with pipelining:

redis-benchmark -r 100000000 -n 100000000  \
        -P 32 set ele:rand:000000000000 foo ex 2

Remove the -P32 in order to avoid the pipelining for a more real-world
load.

In another terminal tab you can monitor the Redis behavior with:

redis-cli -i 0.1 -r -1 info keyspace

and

redis-cli --latency-history

Note: this commit will make Redis printing a lot of debug messages, it
is not a good idea to use it in production.
b09ea1b
@antirez antirez referenced this issue from a commit
@antirez Draft #1 of a new expired keys collection algorithm.
The main idea here is that when we are no longer to expire keys at the
rate the are created, we can't block more in the normal expire cycle as
this would result in too big latency spikes.

For this reason the commit introduces a "fast" expire cycle that does
not run for more than 1 millisecond but is called in the beforeSleep()
hook of the event loop, so much more often, and with a frequency bound
to the frequency of executed commnads.

The fast expire cycle is only called when the standard expiration
algorithm runs out of time, that is, consumed more than
REDIS_EXPIRELOOKUPS_TIME_PERC of CPU in a given cycle without being able
to take the number of already expired keys that are yet not collected
to a number smaller than 25% of the number of keys.

You can test this commit with different loads, but a simple way is to
use the following:

Extreme load with pipelining:

redis-benchmark -r 100000000 -n 100000000  \
        -P 32 set ele:rand:000000000000 foo ex 2

Remove the -P32 in order to avoid the pipelining for a more real-world
load.

In another terminal tab you can monitor the Redis behavior with:

redis-cli -i 0.1 -r -1 info keyspace

and

redis-cli --latency-history

Note: this commit will make Redis printing a lot of debug messages, it
is not a good idea to use it in production.
500155b
@antirez antirez referenced this issue from a commit
@antirez Draft #1 of a new expired keys collection algorithm.
The main idea here is that when we are no longer to expire keys at the
rate the are created, we can't block more in the normal expire cycle as
this would result in too big latency spikes.

For this reason the commit introduces a "fast" expire cycle that does
not run for more than 1 millisecond but is called in the beforeSleep()
hook of the event loop, so much more often, and with a frequency bound
to the frequency of executed commnads.

The fast expire cycle is only called when the standard expiration
algorithm runs out of time, that is, consumed more than
REDIS_EXPIRELOOKUPS_TIME_PERC of CPU in a given cycle without being able
to take the number of already expired keys that are yet not collected
to a number smaller than 25% of the number of keys.

You can test this commit with different loads, but a simple way is to
use the following:

Extreme load with pipelining:

redis-benchmark -r 100000000 -n 100000000  \
        -P 32 set ele:rand:000000000000 foo ex 2

Remove the -P32 in order to avoid the pipelining for a more real-world
load.

In another terminal tab you can monitor the Redis behavior with:

redis-cli -i 0.1 -r -1 info keyspace

and

redis-cli --latency-history

Note: this commit will make Redis printing a lot of debug messages, it
is not a good idea to use it in production.
ab78261
@antirez antirez referenced this issue from a commit
@antirez SDIFF iterator misuse fixed in diff algorithm #1.
The bug could be easily triggered by:

    SADD foo a b c 1 2 3 4 5 6
    SDIFF foo foo

When the key was the same in two sets, an unsafe iterator was used to
check existence of elements in the same set we were iterating.
Usually this would just result into a wrong output, however with the
dict.c API misuse protection we have in place, the result was actually
an assertion failed that was triggered by the CI test, while creating
random datasets for the "MASTER and SLAVE consistency" test.
c00453d
@antirez antirez referenced this issue from a commit
@antirez SDIFF iterator misuse fixed in diff algorithm #1.
The bug could be easily triggered by:

    SADD foo a b c 1 2 3 4 5 6
    SDIFF foo foo

When the key was the same in two sets, an unsafe iterator was used to
check existence of elements in the same set we were iterating.
Usually this would just result into a wrong output, however with the
dict.c API misuse protection we have in place, the result was actually
an assertion failed that was triggered by the CI test, while creating
random datasets for the "MASTER and SLAVE consistency" test.
993e0ed
@sergeyklay sergeyklay referenced this issue from a commit in sergeyklay/redis
@sergeyklay sergeyklay getting pid fixes
```sh
$ ~ pidof redis-server
# nothing

$ ~ ps aux | grep [r]edis
redis      593  0.0  0.0  36900  5564 ?        Ssl  Dec02   1:37 /usr/bin/redis-server 127.0.0.1:6379
klay     15927  0.0  0.0  16772  6068 pts/6    S+   13:58   0:00 redis-cli

$ ~ uname -a
Linux edge 3.17.4-1-ARCH #1 SMP PREEMPT Fri Nov 21 21:14:42 CET 2014 x86_64 GNU/Linux
```
d4a9836
@antirez antirez referenced this issue from a commit
@sergeyklay sergeyklay getting pid fixes
```sh
$ ~ pidof redis-server
# nothing

$ ~ ps aux | grep [r]edis
redis      593  0.0  0.0  36900  5564 ?        Ssl  Dec02   1:37 /usr/bin/redis-server 127.0.0.1:6379
klay     15927  0.0  0.0  16772  6068 pts/6    S+   13:58   0:00 redis-cli

$ ~ uname -a
Linux edge 3.17.4-1-ARCH #1 SMP PREEMPT Fri Nov 21 21:14:42 CET 2014 x86_64 GNU/Linux
```
b5737d2
@antirez antirez referenced this issue from a commit
@sergeyklay sergeyklay getting pid fixes
```sh
$ ~ pidof redis-server
# nothing

$ ~ ps aux | grep [r]edis
redis      593  0.0  0.0  36900  5564 ?        Ssl  Dec02   1:37 /usr/bin/redis-server 127.0.0.1:6379
klay     15927  0.0  0.0  16772  6068 pts/6    S+   13:58   0:00 redis-cli

$ ~ uname -a
Linux edge 3.17.4-1-ARCH #1 SMP PREEMPT Fri Nov 21 21:14:42 CET 2014 x86_64 GNU/Linux
```
81eed2f
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.