Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"config get repl-backlog-size huge-numbers" makes redis crash #6043

Open
JeremyYu2018 opened this issue Apr 24, 2019 · 1 comment
Open

"config get repl-backlog-size huge-numbers" makes redis crash #6043

JeremyYu2018 opened this issue Apr 24, 2019 · 1 comment

Comments

@JeremyYu2018
Copy link

JeremyYu2018 commented Apr 24, 2019

redis version 3.0.7 belows are key logs

33399:M 21 Apr 09:17:18.242 * Unable to partial resync with slave x.x.x.x:6379 for lack of backlog (Slave request was: 245114654245741).
33399:M 21 Apr 09:17:18.242 * Starting BGSAVE for SYNC with target: disk
33399:M 21 Apr 09:17:18.433 * Background saving started by pid 31246
33399:M 21 Apr 09:17:52.857 # Connection with slave x.x.x.x:6379 lost.
31246:C 21 Apr 09:18:10.796 * DB saved on disk
31246:C 21 Apr 09:18:11.058 * RDB: 1082 MB of memory used by copy-on-write
33399:M 21 Apr 09:18:11.585 * Background saving terminated with success
33399:M 21 Apr 09:24:20.958 # Out Of Memory allocating 545114654245741 bytes!
33399:M 21 Apr 09:24:20.958 # 

=== REDIS BUG REPORT START: Cut & paste starting from here ===
33399:M 21 Apr 09:24:20.958 # ------------------------------------------------
33399:M 21 Apr 09:24:20.958 # !!! Software Failure. Press left mouse button to continue
33399:M 21 Apr 09:24:20.958 # Guru Meditation: "Redis aborting for OUT OF MEMORY" #redis.c:3572
33399:M 21 Apr 09:24:20.958 # (forcing SIGSEGV in order to print the stack trace)
33399:M 21 Apr 09:24:20.958 # ------------------------------------------------
33399:M 21 Apr 09:24:20.958 #     Redis 3.0.7 crashed by signal: 11
33399:M 21 Apr 09:24:20.958 #     SIGSEGV caused by address: 0xffffffffffffffff
33399:M 21 Apr 09:24:20.958 #     Failed assertion: <no assertion failed> (<no file>:0)
33399:M 21 Apr 09:24:20.958 # --- STACK TRACE
/usr/local/bin/redis-server *:6379(logStackTrace+0x43)[0x450aa3]
/usr/local/bin/redis-server *:6379(_redisPanic+0x7f)[0x4505bf]
/lib64/libpthread.so.0[0x327dc0f7e0]
/usr/local/bin/redis-server *:6379(_redisPanic+0x7f)[0x4505bf]
/usr/local/bin/redis-server *:6379(redisOutOfMemoryHandler+0x2c)[0x4224ac]
/usr/local/bin/redis-server *:6379(zmalloc+0x71)[0x425a41]
/usr/local/bin/redis-server *:6379(resizeReplicationBacklog+0x41)[0x4356e1]
/usr/local/bin/redis-server *:6379(configSetCommand+0x12a9)[0x44a079]
/usr/local/bin/redis-server *:6379(call+0x72)[0x420292]

33399:M 21 Apr 09:24:21.039 # --- CURRENT CLIENT INFO
33399:M 21 Apr 09:24:21.039 # client: id=574559186 addr=127.0.0.1:57989 fd=859 name= age=105 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=config
33399:M 21 Apr 09:24:21.039 # argv[0]: 'config'
33399:M 21 Apr 09:24:21.039 # argv[1]: 'set'
33399:M 21 Apr 09:24:21.039 # argv[2]: 'repl-backlog-size'
33399:M 21 Apr 09:24:21.039 # argv[3]: '545114654245741'
33399:M 21 Apr 09:24:21.039 # --- REGISTERS
33399:M 21 Apr 09:24:21.039 # 
RAX:0000000000000000 RBX:00000000004bdab0
RCX:0000000000000000 RDX:0000000000014460
RDI:000000327db8fe80 RSI:0000000000000000
RBP:00000000004cc887 RSP:00007ffc94ff8010
R8 :000000327db8fe80 R9 :0000000000000000
R10:000000327db8fed0 R11:0000000000000206
R12:0000000000000df4 R13:0000000000004000
R14:000000000008f0f0 R15:000587003617cc08
RIP:00000000004505bf EFL:0000000000010202
CSGSFS:2d2d000000000033
33399:M 21 Apr 09:24:21.039 # (00007ffc94ff801f) -> 00007ffc94fff8bc
33399:M 21 Apr 09:24:21.039 # (00007ffc94ff801e) -> 00007ffc94ff80a0
33399:M 21 Apr 09:24:21.039 # (00007ffc94ff801d) -> 0000000000000000
33399:M 21 Apr 09:24:21.039 # (00007ffc94ff801c) -> 00007ffc94ff80c0
33399:M 21 Apr 09:24:21.040 # (00007ffc94ff801b) -> 000000000044a079
33399:M 21 Apr 09:24:21.040 # (00007ffc94ff801a) -> 000000000008f0f0
33399:M 21 Apr 09:24:21.040 # (00007ffc94ff8019) -> 00000000004356e1
33399:M 21 Apr 09:24:21.040 # (00007ffc94ff8018) -> 00007f3387f9f000
33399:M 21 Apr 09:24:21.040 # (00007ffc94ff8017) -> 00007f35a78b07c0
33399:M 21 Apr 09:24:21.040 # (00007ffc94ff8016) -> 00007f35aba40e68
33399:M 21 Apr 09:24:21.040 # (00007ffc94ff8015) -> 0000000000425a41
33399:M 21 Apr 09:24:21.040 # (00007ffc94ff8014) -> 000587003617cc08
33399:M 21 Apr 09:24:21.040 # (00007ffc94ff8013) -> 00000000004224ac

This could also happen to redis 4.0.8, and I think it's a dangerous synonym in some scenarios。 so can we just make it safer?

@AngusP
Copy link
Contributor

AngusP commented Apr 24, 2019

If you ask Redis to allocate you a massive buffer, there's not a huge amount really wrong with it trying 😉 - however, failing in a slightly nicer manner would perhaps be good, rather than OOMing. Remember that CONFIG is a privileged command, there are many ways to cause breakage with it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants