export going up to 280% or more

[sirkubax]

v 1.9.4

Error probably to faulty DB size estimation, and wrong export status presentation, due to compressed leveldb database.
For me it is a bug - the database should present the proper export %, or it should have an option to export the content in the compressed form (then the % would be correct)

The export is running for about 280%

I did give up when the exported file was ~120GB size, 220% (the DB size was about 56GB).
What is going wrong?

ssdb 127.0.0.1:8888> export -i /data/ssdb.20170122.ssdbexport
input KV range[start, end]:
  start(inclusive, default none):
    end(inclusive, default none):
input HASH range:
  start(inclusive, default none):
    end(inclusive, default none):
input ZSET range:
  start(inclusive, default none):
    end(inclusive, default none):
input QUEUE range:
  start(inclusive, default none):
    end(inclusive, default none):

 5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
105%
110%
115%
120%
...
220% << I'm aborting here

It may be just counting error, but still - if the exported data takes 2-3x the DB size (probably compressed), then it is not so good.

The other time I did export in the past (the DB was about 2GB, the export file about 7GB)

ssdb 127.0.0.1:8888> export -i /home/ubuntu/ssdb.dump.20161223.00.09
input KV range[start, end]:
  start(inclusive, default none):
    end(inclusive, default none):
input HASH range:
  start(inclusive, default none):
    end(inclusive, default none):
input ZSET range:
  start(inclusive, default none):
    end(inclusive, default none):
input QUEUE range:
  start(inclusive, default none):
    end(inclusive, default none):
 5% 
10%
15%
20%
25%
30%
35%
40%
45%
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
105%
110%
115%
(..)
225%
230%
235%
240%
245%
250%
255%
260%
265%
270%
275%
280% << I'm aborting here

Similar bug
#930

[henpa]

Hi.. I can also easily replicate this behavior.

1. I started with a fresh and clean ssdb-server:

ssdb 127.0.0.1:8888> info
version
        1.9.4
links
        1
total_calls
        1
dbsize
        0
binlogs
            capacity : 20000000
            min_seq  : 0
            max_seq  : 0
serv_key_range
            kv  : "" - ""
            hash: "" - ""
            zset: "" - ""
            list: "" - ""
data_key_range
            kv  : "" - ""
            hash: "" - ""
            zset: "" - ""
            list: "" - ""
leveldb.stats
                                       Compactions
        Level  Files Size(MB) Time(sec) Read(MB) Write(MB)
        --------------------------------------------------

17 result(s) (0.000 sec)
(0.000 sec)

2. then I inserted some data

ssdb 127.0.0.1:8888> info
version
        1.9.4
links
        1
total_calls
        669954
dbsize
        12313499
binlogs
            capacity : 20000000
            min_seq  : 0
            max_seq  : 669952
serv_key_range
            kv  : "" - ""
            hash: "" - ""
            zset: "" - ""
            list: "" - ""
data_key_range
            kv  : "" - ""
            hash: "envio:111:1" - "envio:9:9"
            zset: "envios:1" - "envios:9"
            list: "" - ""
leveldb.stats
                                       Compactions
        Level  Files Size(MB) Time(sec) Read(MB) Write(MB)
        --------------------------------------------------
          1        0        0         0        0        12
          2        1       20         2       24        33

17 result(s) (0.001 sec)
(0.001 sec)

3. then I exported

ssdb 127.0.0.1:8888> export /tmp/1.ssdb
 5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
105%
110%
115%
120%
125%
130%
135%
140%
145%
150%
155%
160%
165%
^C << I aborted here

What's going on? Is ssdb production-ready?

Thank you!

[sirkubax]

@henpa
I guess you have compression: yes set in the config. Then it might be difficult to estimate the size of the DB. @ideawu I'd consider a change in % presentation for the compressed DB.

Additionally, it would be nice to warn a user, that his DB dump might be 3-5x the size of the DB presented in the info. In my case, I had 56GB DB, with 190GB disk space at this time - so I had to abort because 220% of the export happened to ~120GB - getting close to the dangerous 100% of this utilisation...

Since the process took a long time, I did actually give up on adding more disk space and repeating the process...

It would be great to give some estimation on how long the export could take (120GB ~ 3-4 hours)...

leveldb:
        # in MB
        cache_size: 2500
        # in KB
        block_size: 32
        # in MB
        write_buffer_size: 128
        # in MB
        compaction_speed: 1000
        # yes|no
        compression: yes

I do consider disabling the compression in the future...

[sirkubax]

@henpa

What's going on? Is ssdb production-ready?

Well, it is the best choice for a redis replacement, that can store the DB on the disk (redis is server-memory limited). You may find other tools like ssdb, but only ssdb is having an ongoing development (as I browse the github commits), and I'd not install a product that's development is discontinued.

It would work for you, unless you have a heavy writes (see #1045) In our case we are going to replace 1 out of 3 SSDB clusters that we have, and switch back to redis, since SSDB can not cope with bulk writes, and is getting OUT_OF_SYNC, that requires manual, few-hours work to fix. @ideawu it really should fix itself...

So is this production ready... Heh - we do use this on production for about 8-10 months, it is working, but maintenance is way higher (due to out_of_sync) that we did expect.

[zhoux2016]

see(#613)This progress is only a reference, the impact of compression is relatively large, even if not compressed, but also affected by the impact of the organization of the data
wait for a long time it will be done

It took 13 hours to export