redis-data-estimation
How much data will Redis use ?
In order to make a better guess, this tools will generate the data and print out the size used in Redis to the terminal.
Questions
- What will be the volume in terms of requests per second
- data amounts to store
- will this be cache only
- is there a requirements for persistence
- how small can this cluster be in staging compared to prd
Requirements
docker
or if running locally:
node
LTS (version: 8+)redis
Environmental Variables
ENV | DESCRIPTION | DEFAULT |
---|---|---|
PREFIX | Prefix for redis keys | test:${random(0, 100)}:${COMMAND} |
COMMAND | Redis command to add data | sadd |
TOTAL_ENTRY | Total number of keys | 100 |
MAX_ELEMENTS | Maximum number of elements/key (if applicable) | 10 |
MIN_ELEMENTS | Minimum number of elements/key (if applicable) | 1 |
BATCH_SIZE | Number of keys to batch insert at once | 100 |
PROGRESS | Display a progress bar | true |
VERBOSE | Print out the first key of each batch | false |
REDIS_HOST | Redis host to connect to | redis |
REDIS_PORT | Redis port to connect to | 6379 |
How to use it
The app will generate random data using lodash/random
for values and crypto.randomBytes
from
the crypto
node modules to generate uuid for keys. An array of commands are batched together and
executed at once. To define the size of the batch use the environmental variable BATCH_SIZE
.
Note that the size of the batch should be smaller than the total number of entry generated.
( TOTAL_ENTRY
>= BATCH_SIZE
)
Redis uses different data structure such as set
, hash
, list
, ... .
For each data structure, there is a command that can add data to it
such SADD
for a set
, HMSET
for a hash
and so on.
To use a specific data structure, define the command by the environmental variable COMMAND
Some commands have more than one parameter.
To define the number of parameters,
use the environmental variable MIN_ELEMENTS
and MAX_ELEMENTS
.
The size of the data in Redis is obtained by using redis-cli
:
redis-cli info memory
Here is an example of the execution:
Run Benchmark in a container
This will spin up 2 containers: redis
and node index.js
(the app).
Simply use npm run start
or directly the bash script ./run.sh
COMMAND=sadd TOTAL_ENTRY=100000 BATCH_SIZE=5000 npm run start
Run Benchmark locally
Just use npm run start:local
COMMAND=lpush TOTAL_ENTRY=100000 BATCH_SIZE=5000 npm run start:local