New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

active memory defragmentation #3720

Merged
merged 2 commits into from Jan 11, 2017

Conversation

Projects
None yet
3 participants
@oranagra
Contributor

oranagra commented Jan 1, 2017

gradually scan the keyspace and all the allocations in it, and re-allocate them in order to reduce memory fragmentation.

notes:

  • we may want to make it disabled by default in the released version, but for now i prefer to keep it enabled so that it can be tested.
  • i've also added an optional size parameter to DEBUG POPULATE
@oranagra

This comment has been minimized.

Show comment
Hide comment
@oranagra

oranagra Jan 1, 2017

Contributor

Demonstration:
$ redis-server --maxmemory 1gb --maxmemory-policy allkeys-lru --activedefrag no
$ redis-cli debug populate 7000000 asdf 150

$ redis-cli info keyspace
db0:keys=4656506,expires=0,avg_ttl=0
$ redis-cli info memory
used_memory:1073741744
used_memory_rss:1621311488
$ redis-cli memory malloc-stats
Allocated: 1074215072, active: 1580675072, metadata: 40734080, resident: 1621250048, mapped: 1623195648
bins:           size ind    allocated      nmalloc      ndalloc    nrequests      curregs      curruns regs pgs  util       nfills     nflushes      newruns       reruns
                  16   1    149174192     14010400      4687013     14010590      9323387        54728  256   1 0.665       140104        47456        54729            0
                  24   2    111771936      7000541      2343377      7000941      4657164        13673  512   3 0.665        70046        24230        13673            4
                 160  12    745070720      7000000      2343308      7000000      4656692        54688  128   5 0.665        70000        24230        54688            3
                 320  16         5120           64           48           16           16            1   64   5 0.250            1            3            1            0

$ redis-cli debug populate 1700000 asdf 300

$ redis-cli info keyspace
db0:keys=4319729,expires=0,avg_ttl=0
$ redis-cli info memory
used_memory:1073741680
used_memory_rss:1770536960
$ redis-cli memory malloc-stats
Allocated: 1074035368, active: 1726078976, metadata: 44567936, resident: 1770532864, mapped: 1774190592
bins:           size ind    allocated      nmalloc      ndalloc    nrequests      curregs      curruns regs pgs  util       nfills     nflushes      newruns       reruns
                  16   1    138398640     14919679      6269764     14920166      8649915        54728  256   1 0.617       151542        63481        54729         8506
                  24   2    103689456      7455030      3134636      7456048      4320394        13673  512   3 0.617        74992        32411        13673         2133
                 160  12    618436480      7000000      3134772      7000008      3865228        54688  128   5 0.552        70000        32416        54688            3
                 320  16    145492160       454712           49       454663       454663         7105   64   5 0.999         7181            4         7105            0

$ redis-cli config set activedefrag yes

$ redis-cli info memory
used_memory:1073741680
used_memory_rss:1254879232
$ redis-cli memory malloc-stats
Allocated: 1074028184, active: 1075724288, metadata: 44567936, resident: 1254559744, mapped: 1774190592
bins:           size ind    allocated      nmalloc      ndalloc    nrequests      curregs      curruns regs pgs  util       nfills     nflushes      newruns       reruns
                  16   1    138398640     19485481     10835566     19485968      8649915        33809  256   1 0.999       151542        63481        54729        42254
                  24   2    103689456      9760235      5439841      9761253      4320394         8440  512   3 0.999        74992        32411        13673        10570
                 160  12    618428960      9618126      5752945      9618141      3865181        30256  128   5 0.998        70000        32417        54688        29967
                 320  16    145492160       454712           49       454663       454663         7105   64   5 0.999         7181            4         7105            0

$ src/redis-cli memory purge

$ redis-cli info memory
used_memory:1073741824
used_memory_rss:1120448512
$ redis-cli memory malloc-stats
Allocated: 1073997440, active: 1075675136, metadata: 44514816, resident: 1122226176, mapped: 1772093440
nfills     nflushes      newruns       reruns
                  16   1    138396816     19485481     10835680     19486538      8649801        33809  256   1 0.999       151542        63482        54729        42258
                  24   2    103689384      9760235      5439844      9762208      4320391         8440  512   3 0.999        74992        32412        13673        10571
                 160  12    618417920      9618126      5753014      9618162      3865112        30256  128   5 0.998        70000        32418        54688        29969
                 320  16    145492160       454712           49       454663       454663         7105   64   5 0.999         7181            4         7105            0
Contributor

oranagra commented Jan 1, 2017

Demonstration:
$ redis-server --maxmemory 1gb --maxmemory-policy allkeys-lru --activedefrag no
$ redis-cli debug populate 7000000 asdf 150

$ redis-cli info keyspace
db0:keys=4656506,expires=0,avg_ttl=0
$ redis-cli info memory
used_memory:1073741744
used_memory_rss:1621311488
$ redis-cli memory malloc-stats
Allocated: 1074215072, active: 1580675072, metadata: 40734080, resident: 1621250048, mapped: 1623195648
bins:           size ind    allocated      nmalloc      ndalloc    nrequests      curregs      curruns regs pgs  util       nfills     nflushes      newruns       reruns
                  16   1    149174192     14010400      4687013     14010590      9323387        54728  256   1 0.665       140104        47456        54729            0
                  24   2    111771936      7000541      2343377      7000941      4657164        13673  512   3 0.665        70046        24230        13673            4
                 160  12    745070720      7000000      2343308      7000000      4656692        54688  128   5 0.665        70000        24230        54688            3
                 320  16         5120           64           48           16           16            1   64   5 0.250            1            3            1            0

$ redis-cli debug populate 1700000 asdf 300

$ redis-cli info keyspace
db0:keys=4319729,expires=0,avg_ttl=0
$ redis-cli info memory
used_memory:1073741680
used_memory_rss:1770536960
$ redis-cli memory malloc-stats
Allocated: 1074035368, active: 1726078976, metadata: 44567936, resident: 1770532864, mapped: 1774190592
bins:           size ind    allocated      nmalloc      ndalloc    nrequests      curregs      curruns regs pgs  util       nfills     nflushes      newruns       reruns
                  16   1    138398640     14919679      6269764     14920166      8649915        54728  256   1 0.617       151542        63481        54729         8506
                  24   2    103689456      7455030      3134636      7456048      4320394        13673  512   3 0.617        74992        32411        13673         2133
                 160  12    618436480      7000000      3134772      7000008      3865228        54688  128   5 0.552        70000        32416        54688            3
                 320  16    145492160       454712           49       454663       454663         7105   64   5 0.999         7181            4         7105            0

$ redis-cli config set activedefrag yes

$ redis-cli info memory
used_memory:1073741680
used_memory_rss:1254879232
$ redis-cli memory malloc-stats
Allocated: 1074028184, active: 1075724288, metadata: 44567936, resident: 1254559744, mapped: 1774190592
bins:           size ind    allocated      nmalloc      ndalloc    nrequests      curregs      curruns regs pgs  util       nfills     nflushes      newruns       reruns
                  16   1    138398640     19485481     10835566     19485968      8649915        33809  256   1 0.999       151542        63481        54729        42254
                  24   2    103689456      9760235      5439841      9761253      4320394         8440  512   3 0.999        74992        32411        13673        10570
                 160  12    618428960      9618126      5752945      9618141      3865181        30256  128   5 0.998        70000        32417        54688        29967
                 320  16    145492160       454712           49       454663       454663         7105   64   5 0.999         7181            4         7105            0

$ src/redis-cli memory purge

$ redis-cli info memory
used_memory:1073741824
used_memory_rss:1120448512
$ redis-cli memory malloc-stats
Allocated: 1073997440, active: 1075675136, metadata: 44514816, resident: 1122226176, mapped: 1772093440
nfills     nflushes      newruns       reruns
                  16   1    138396816     19485481     10835680     19486538      8649801        33809  256   1 0.999       151542        63482        54729        42258
                  24   2    103689384      9760235      5439844      9762208      4320391         8440  512   3 0.999        74992        32412        13673        10571
                 160  12    618417920      9618126      5753014      9618162      3865112        30256  128   5 0.998        70000        32418        54688        29969
                 320  16    145492160       454712           49       454663       454663         7105   64   5 0.999         7181            4         7105            0
@itamarhaber

This comment has been minimized.

Show comment
Hide comment
@itamarhaber

itamarhaber Jan 4, 2017

Contributor

This will make so many users so happy

Contributor

itamarhaber commented Jan 4, 2017

This will make so many users so happy

@antirez

This comment has been minimized.

Show comment
Hide comment
@antirez

antirez Jan 9, 2017

Owner

Thanks Oran! I'm reviewing this today.

Owner

antirez commented Jan 9, 2017

Thanks Oran! I'm reviewing this today.

@antirez antirez merged commit 5ab6a54 into antirez:unstable Jan 11, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment