VM can't suppert Lists Or sets have much numbers。 #1

Closed
tongxiaojun opened this Issue Jan 23, 2010 · 6 comments

Projects

None yet

2 participants

VM can't suppert Lists Or sets have much numbers。

Owner
antirez commented Jan 23, 2010

Hello,

can you please elaborate a bit? Like how much numbers, how to replicate, and the error reported by Redis. Thanks!

when I use command like redis-cli -r 10000000 lpush mylist aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa[1k content]。the redis will use a lot of memory。

redis_version:1.3.3
arch_bits:64
multiplexing_api:epoll
process_id:26560
uptime_in_seconds:106
uptime_in_days:0
connected_clients:2
connected_slaves:0
blocked_clients:0
used_memory:179048675
used_memory_human:170.75M
changes_since_last_save:559778
bgsave_in_progress:0
last_save_time:1264264099
bgrewriteaof_in_progress:0
total_connections_received:39
total_commands_processed:1181478
vm_enabled:1
role:master
vm_conf_max_memory:0
vm_conf_page_size:256
vm_conf_pages:104857600
vm_stats_used_pages:0
vm_stats_swapped_objects:0
vm_stats_swappin_count:1
vm_stats_swappout_count:1
vm_stats_io_newjobs_len:0
vm_stats_io_processing_len:0
vm_stats_io_processed_len:0
vm_stats_io_waiting_clients:0
vm_stats_io_active_threads:0
db0:keys=1,expires=0

this info show redis can't VM one key。

my redis config is vm-max-memory 0
I want use reids list pop push make a simple queue 。
but I find redis can't VM a key lists to many numbers。like 1000 0000 numbers and use a little memory。and it will use nearby 80% system memory at most。and when i run this long time 。
the linux will crash [my max system memory is 4G ]。

Owner
antirez commented Jan 23, 2010

Hello, thanks for the info, could you please enter IRC irc.freenode.org #redis for a chat?

Owner
antirez commented Feb 19, 2010

Can't remember the details, but this issue is now closed.

@antirez antirez added a commit that referenced this issue Jan 8, 2013
@antirez antirez PSYNC: work in progress, preview #1 6f6ba62
@antirez antirez added a commit that referenced this issue Aug 5, 2013
@antirez antirez Draft #1 of a new expired keys collection algorithm.
The main idea here is that when we are no longer to expire keys at the
rate the are created, we can't block more in the normal expire cycle as
this would result in too big latency spikes.

For this reason the commit introduces a "fast" expire cycle that does
not run for more than 1 millisecond but is called in the beforeSleep()
hook of the event loop, so much more often, and with a frequency bound
to the frequency of executed commnads.

The fast expire cycle is only called when the standard expiration
algorithm runs out of time, that is, consumed more than
REDIS_EXPIRELOOKUPS_TIME_PERC of CPU in a given cycle without being able
to take the number of already expired keys that are yet not collected
to a number smaller than 25% of the number of keys.

You can test this commit with different loads, but a simple way is to
use the following:

Extreme load with pipelining:

redis-benchmark -r 100000000 -n 100000000  \
        -P 32 set ele:rand:000000000000 foo ex 2

Remove the -P32 in order to avoid the pipelining for a more real-world
load.

In another terminal tab you can monitor the Redis behavior with:

redis-cli -i 0.1 -r -1 info keyspace

and

redis-cli --latency-history

Note: this commit will make Redis printing a lot of debug messages, it
is not a good idea to use it in production.
b09ea1b
@antirez antirez added a commit that referenced this issue Aug 6, 2013
@antirez antirez Draft #1 of a new expired keys collection algorithm.
The main idea here is that when we are no longer to expire keys at the
rate the are created, we can't block more in the normal expire cycle as
this would result in too big latency spikes.

For this reason the commit introduces a "fast" expire cycle that does
not run for more than 1 millisecond but is called in the beforeSleep()
hook of the event loop, so much more often, and with a frequency bound
to the frequency of executed commnads.

The fast expire cycle is only called when the standard expiration
algorithm runs out of time, that is, consumed more than
REDIS_EXPIRELOOKUPS_TIME_PERC of CPU in a given cycle without being able
to take the number of already expired keys that are yet not collected
to a number smaller than 25% of the number of keys.

You can test this commit with different loads, but a simple way is to
use the following:

Extreme load with pipelining:

redis-benchmark -r 100000000 -n 100000000  \
        -P 32 set ele:rand:000000000000 foo ex 2

Remove the -P32 in order to avoid the pipelining for a more real-world
load.

In another terminal tab you can monitor the Redis behavior with:

redis-cli -i 0.1 -r -1 info keyspace

and

redis-cli --latency-history

Note: this commit will make Redis printing a lot of debug messages, it
is not a good idea to use it in production.
500155b
@antirez antirez added a commit that referenced this issue Aug 7, 2013
@antirez antirez Draft #1 of a new expired keys collection algorithm.
The main idea here is that when we are no longer to expire keys at the
rate the are created, we can't block more in the normal expire cycle as
this would result in too big latency spikes.

For this reason the commit introduces a "fast" expire cycle that does
not run for more than 1 millisecond but is called in the beforeSleep()
hook of the event loop, so much more often, and with a frequency bound
to the frequency of executed commnads.

The fast expire cycle is only called when the standard expiration
algorithm runs out of time, that is, consumed more than
REDIS_EXPIRELOOKUPS_TIME_PERC of CPU in a given cycle without being able
to take the number of already expired keys that are yet not collected
to a number smaller than 25% of the number of keys.

You can test this commit with different loads, but a simple way is to
use the following:

Extreme load with pipelining:

redis-benchmark -r 100000000 -n 100000000  \
        -P 32 set ele:rand:000000000000 foo ex 2

Remove the -P32 in order to avoid the pipelining for a more real-world
load.

In another terminal tab you can monitor the Redis behavior with:

redis-cli -i 0.1 -r -1 info keyspace

and

redis-cli --latency-history

Note: this commit will make Redis printing a lot of debug messages, it
is not a good idea to use it in production.
ab78261
@antirez antirez added a commit that referenced this issue Dec 13, 2013
@antirez antirez SDIFF iterator misuse fixed in diff algorithm #1.
The bug could be easily triggered by:

    SADD foo a b c 1 2 3 4 5 6
    SDIFF foo foo

When the key was the same in two sets, an unsafe iterator was used to
check existence of elements in the same set we were iterating.
Usually this would just result into a wrong output, however with the
dict.c API misuse protection we have in place, the result was actually
an assertion failed that was triggered by the CI test, while creating
random datasets for the "MASTER and SLAVE consistency" test.
c00453d
@antirez antirez added a commit that referenced this issue Dec 13, 2013
@antirez antirez SDIFF iterator misuse fixed in diff algorithm #1.
The bug could be easily triggered by:

    SADD foo a b c 1 2 3 4 5 6
    SDIFF foo foo

When the key was the same in two sets, an unsafe iterator was used to
check existence of elements in the same set we were iterating.
Usually this would just result into a wrong output, however with the
dict.c API misuse protection we have in place, the result was actually
an assertion failed that was triggered by the CI test, while creating
random datasets for the "MASTER and SLAVE consistency" test.
993e0ed
@antirez antirez added a commit that referenced this issue Dec 9, 2014
@sergeyklay @antirez sergeyklay + antirez getting pid fixes
```sh
$ ~ pidof redis-server
# nothing

$ ~ ps aux | grep [r]edis
redis      593  0.0  0.0  36900  5564 ?        Ssl  Dec02   1:37 /usr/bin/redis-server 127.0.0.1:6379
klay     15927  0.0  0.0  16772  6068 pts/6    S+   13:58   0:00 redis-cli

$ ~ uname -a
Linux edge 3.17.4-1-ARCH #1 SMP PREEMPT Fri Nov 21 21:14:42 CET 2014 x86_64 GNU/Linux
```
b5737d2
@antirez antirez added a commit that referenced this issue Dec 9, 2014
@sergeyklay @antirez sergeyklay + antirez getting pid fixes
```sh
$ ~ pidof redis-server
# nothing

$ ~ ps aux | grep [r]edis
redis      593  0.0  0.0  36900  5564 ?        Ssl  Dec02   1:37 /usr/bin/redis-server 127.0.0.1:6379
klay     15927  0.0  0.0  16772  6068 pts/6    S+   13:58   0:00 redis-cli

$ ~ uname -a
Linux edge 3.17.4-1-ARCH #1 SMP PREEMPT Fri Nov 21 21:14:42 CET 2014 x86_64 GNU/Linux
```
81eed2f
@antirez antirez added a commit that referenced this issue Sep 23, 2015
@antirez antirez Threaded lazyfree WIP #1. ddc6abd
@antirez antirez added a commit that referenced this issue Oct 2, 2015
@antirez antirez Threaded lazyfree WIP #1. 9253d85
@thepirat000 thepirat000 referenced this issue Mar 2, 2016
@antirez antirez BITFIELD command initial implementation.
The new bitfield command is an extension to the Redis bit operations,
where not just single bit operations are performed, but the array of
bits composing a string, can be addressed at random, not aligned
offsets, with any width unsigned and signed integers like u8, s5, u10
(up to 64 bit signed integers and 63 bit unsigned integers).

The BITFIELD command supports subcommands that can SET, GET, or INCRBY
those arbitrary bit counters, with multiple overflow semantics.

Trivial and credits:

A similar command was imagined a few times in the past, but for
some reason looked a bit far fetched or not well specified.
Finally the command was proposed again in a clear form by
Yoav Steinberg from Redis Labs, that proposed a set of commands on
arbitrary sized integers stored at bit offsets.

Starting from this proposal I wrote an initial specification of a single
command with sub-commands similar to what Yoav envisioned, using short
names for types definitions, and adding control on the overflow.

This commit is the resulting implementation.

Examples:

    BITFIELD mykey OVERFLOW wrap INCRBY i2 10 -1 GET i2 10
70af626
@antirez antirez added a commit that referenced this issue May 10, 2016
@antirez antirez Modules: zset lex iterator #1. 2b04f86
@antirez antirez added a commit that referenced this issue May 10, 2016
@antirez antirez Modules: Hash type API WIP #1. 10993ca
@JackieXie168 JackieXie168 pushed a commit to JackieXie168/redis that referenced this issue Aug 29, 2016
@antirez antirez Draft #1 of a new expired keys collection algorithm.
The main idea here is that when we are no longer to expire keys at the
rate the are created, we can't block more in the normal expire cycle as
this would result in too big latency spikes.

For this reason the commit introduces a "fast" expire cycle that does
not run for more than 1 millisecond but is called in the beforeSleep()
hook of the event loop, so much more often, and with a frequency bound
to the frequency of executed commnads.

The fast expire cycle is only called when the standard expiration
algorithm runs out of time, that is, consumed more than
REDIS_EXPIRELOOKUPS_TIME_PERC of CPU in a given cycle without being able
to take the number of already expired keys that are yet not collected
to a number smaller than 25% of the number of keys.

You can test this commit with different loads, but a simple way is to
use the following:

Extreme load with pipelining:

redis-benchmark -r 100000000 -n 100000000  \
        -P 32 set ele:rand:000000000000 foo ex 2

Remove the -P32 in order to avoid the pipelining for a more real-world
load.

In another terminal tab you can monitor the Redis behavior with:

redis-cli -i 0.1 -r -1 info keyspace

and

redis-cli --latency-history

Note: this commit will make Redis printing a lot of debug messages, it
is not a good idea to use it in production.
83995f1
@JackieXie168 JackieXie168 pushed a commit to JackieXie168/redis that referenced this issue Aug 29, 2016
@antirez antirez RSS aware maxmemory: algorith #1 implemented. 22b7269
@JackieXie168 JackieXie168 pushed a commit to JackieXie168/redis that referenced this issue Aug 29, 2016
@antirez antirez Modules: zset iterator redesign #1. 287171b
@JackieXie168 JackieXie168 pushed a commit to JackieXie168/redis that referenced this issue Aug 29, 2016
@antirez antirez Modules: zset lex iterator #1. cf0d9e4
@JackieXie168 JackieXie168 pushed a commit to JackieXie168/redis that referenced this issue Aug 29, 2016
@antirez antirez Modules: Hash type API WIP #1. bb09e2e
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment