Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CRASH] Redis 6.2.7 crashed by signal: 11, si_code: 128 #13137

Open
wangbin77 opened this issue Mar 14, 2024 · 3 comments
Open

[CRASH] Redis 6.2.7 crashed by signal: 11, si_code: 128 #13137

wangbin77 opened this issue Mar 14, 2024 · 3 comments

Comments

@wangbin77
Copy link

wangbin77 commented Mar 14, 2024

Crash report

=== REDIS BUG REPORT START: Cut & paste starting from here ===
928:M 14 Mar 2024 05:10:17.979 # Redis 6.2.7 crashed by signal: 11, si_code: 128
928:M 14 Mar 2024 05:10:17.979 # Accessing address: (nil)
928:M 14 Mar 2024 05:10:17.979 # Crashed running the instruction at: 0x55dae6e57a64

------ STACK TRACE ------
EIP:
/usr/bin/redis-server *:6379(sdsAllocSize+0x4)[0x55dae6e57a64]

Backtrace:
/lib64/libpthread.so.0(+0x141d0)[0x7f8b4ba6e1d0]
/usr/bin/redis-server *:6379(sdsAllocSize+0x4)[0x55dae6e57a64]
/usr/bin/redis-server *:6379(clientsCronResizeQueryBuffer+0x11)[0x55dae6e4d571]
/usr/bin/redis-server *:6379(clientsCron+0xcf)[0x55dae6e4d8bf]
/usr/bin/redis-server *:6379(serverCron+0x15d)[0x55dae6e5689d]
/usr/bin/redis-server *:6379(aeProcessEvents+0x12d)[0x55dae6e48a5d]
/usr/bin/redis-server *:6379(aeMain+0x1d)[0x55dae6e48e9d]
/usr/bin/redis-server *:6379(main+0x337)[0x55dae6e45517]
/lib64/libc.so.6(__libc_start_main+0xf2)[0x7f8b4b462a72]
/usr/bin/redis-server *:6379(_start+0x2e)[0x55dae6e45a5e]

------ REGISTERS ------
928:M 14 Mar 2024 05:10:17.980 #
RAX:0000000000000000 RBX:00007f8b3a8d4280
RCX:0000000000000000 RDX:00007f8b3a99c320
RDI:0500090100030002 RSI:0000018e39a6d27b
RBP:000055dae7014da0 RSP:00007fff6e36a308
R8 :0000000000000010 R9 :0000000000000000
R10:0000000002fd7eb0 R11:000055dae6fc40c0
R12:0000018e39a6d27b R13:0000000000000001
R14:00007f8b3a8d4280 R15:0000000000000000
RIP:000055dae6e57a64 EFL:0000000000010246
CSGSFS:002b000000000033
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a317) -> 0000000000000001
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a316) -> 0000000000000001
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a315) -> 00007fff6e36a3b0
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a314) -> 00007f8b4ae230a0
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a313) -> 00007fff6e3f2a87
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a312) -> 159dd1c100000000
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a311) -> 0000000000000000
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a310) -> 000055dae6e5689d
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a30f) -> 0000000000000001
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a30e) -> 0000000000000001
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a30d) -> 000000000df82c91
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a30c) -> 000000002b11252b
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a30b) -> 000055dae7014da0
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a30a) -> 000055dae6e4d8bf
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a309) -> 000000000000000b
928:M 14 Mar 2024 05:10:17.980 # (00007fff6e36a308) -> 000055dae6e4d571

------ INFO OUTPUT ------

Server

redis_version:6.2.7
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:b61ed404cacc6c27
redis_mode:standalone
os:Linux 5.10.134-16.al8.x86_64 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:10.2.1
process_id:928
process_supervised:systemd
run_id:30afb8da835f31aea9e148b8e4c7163bfef6fac7
tcp_port:6379
server_time_usec:1710364217979570
uptime_in_seconds:25131
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:15865401
executable:/usr/bin/redis-server
config_file:/etc/redis.conf
io_threads_active:0

Clients

connected_clients:226
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:56
client_recent_max_output_buffer:15620604899393263920
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

Memory

used_memory:10228296
used_memory_human:9.75M
used_memory_rss:22315008
used_memory_rss_human:21.28M
used_memory_peak:17536984
used_memory_peak_human:16.72M
used_memory_peak_perc:58.32%
used_memory_overhead:3196599359611562144
used_memory_startup:812112
used_memory_dataset:15250144714108217768
used_memory_dataset_perc:161956740726784.00%
allocator_allocated:10643808
allocator_active:12763136
allocator_resident:21626880
total_system_memory:32844468224
total_system_memory_human:30.59G
used_memory_lua:36864
used_memory_lua_human:36.00K
used_memory_scripts:1344
used_memory_scripts_human:1.31K
number_of_cached_scripts:4
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.20
allocator_frag_bytes:2119328
allocator_rss_ratio:1.69
allocator_rss_bytes:8863744
rss_overhead_ratio:1.03
rss_overhead_bytes:688128
mem_fragmentation_ratio:2.18
mem_fragmentation_bytes:12088504
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:3196599359608774096
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

Persistence

loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:402
rdb_bgsave_in_progress:0
rdb_last_save_time:1710364069
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:749568
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

Stats

total_connections_received:25090
total_commands_processed:2712691
instantaneous_ops_per_sec:31
total_net_input_bytes:234368145
total_net_output_bytes:722543915
instantaneous_input_kbps:0.85
instantaneous_output_kbps:2.35
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:24989
expired_stale_perc:0.41
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:1109
evicted_keys:0
keyspace_hits:381132
keyspace_misses:129591
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:443
total_forks:83
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:120527
dump_payload_sanitizations:0
total_reads_processed:2733415
total_writes_processed:2709074
io_threaded_reads_processed:0
io_threaded_writes_processed:0

Replication

role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:7ef692ab0b1ca1beabc84fd45b2f680e3865f1ad
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:34.274030
used_cpu_user:46.890963
used_cpu_sys_children:0.338626
used_cpu_user_children:2.697668
used_cpu_sys_main_thread:34.140946
used_cpu_user_main_thread:46.806058

Modules

Commandstats

cmdstat_psetex:calls=48525,usec=287200,usec_per_call=5.92,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=902,usec=3944,usec_per_call=4.37,rejected_calls=0,failed_calls=0
cmdstat_lrem:calls=360,usec=2561,usec_per_call=7.11,rejected_calls=0,failed_calls=0
cmdstat_rpush:calls=705,usec=4233,usec_per_call=6.00,rejected_calls=0,failed_calls=0
cmdstat_subscribe:calls=0,usec=0,usec_per_call=0.00,rejected_calls=24187,failed_calls=0
cmdstat_ping:calls=250609,usec=160721,usec_per_call=0.64,rejected_calls=23779,failed_calls=0
cmdstat_publish:calls=3913,usec=3656,usec_per_call=0.93,rejected_calls=24186,failed_calls=0
cmdstat_exists:calls=41800,usec=62376,usec_per_call=1.49,rejected_calls=0,failed_calls=0
cmdstat_del:calls=35850,usec=111341,usec_per_call=3.11,rejected_calls=0,failed_calls=0
cmdstat_llen:calls=37200,usec=62764,usec_per_call=1.69,rejected_calls=0,failed_calls=0
cmdstat_client:calls=0,usec=0,usec_per_call=0.00,rejected_calls=24188,failed_calls=0
cmdstat_pexpire:calls=13606,usec=25953,usec_per_call=1.91,rejected_calls=0,failed_calls=0
cmdstat_scan:calls=1804958,usec=11959132,usec_per_call=6.63,rejected_calls=0,failed_calls=0
cmdstat_incr:calls=1900,usec=9648,usec_per_call=5.08,rejected_calls=0,failed_calls=0
cmdstat_sismember:calls=1563,usec=3389,usec_per_call=2.17,rejected_calls=0,failed_calls=0
cmdstat_get:calls=426247,usec=1485158,usec_per_call=3.48,rejected_calls=0,failed_calls=0
cmdstat_incrby:calls=7402,usec=22589,usec_per_call=3.05,rejected_calls=0,failed_calls=0
cmdstat_info:calls=0,usec=0,usec_per_call=0.00,rejected_calls=24187,failed_calls=0
cmdstat_hincrby:calls=7826,usec=37670,usec_per_call=4.81,rejected_calls=0,failed_calls=0
cmdstat_sadd:calls=4,usec=136,usec_per_call=34.00,rejected_calls=0,failed_calls=0
cmdstat_hexists:calls=3913,usec=14625,usec_per_call=3.74,rejected_calls=0,failed_calls=0
cmdstat_eval:calls=25055,usec=779315,usec_per_call=31.10,rejected_calls=0,failed_calls=0
cmdstat_lpop:calls=353,usec=3064,usec_per_call=8.68,rejected_calls=0,failed_calls=0

Errorstats

errorstat_NOAUTH:count=120527

Cluster

cluster_enabled:0

Keyspace

db0:keys=22661,expires=22661,avg_ttl=1558842669

Additional information

  1. OS distribution and version
    Linux version 5.10.134-16.al8.x86_64 (mockbuild@a12i03132.na61) (gcc (GCC) 10.2.1 20200825 (Alibaba 10.2.1-3.5 2.32), GNU ld version 2.35-12.2.al8) VM can't suppert Lists Or sets have much numbers。 #1 SMP Thu Nov 16 05:08:49 UTC 2023

  2. Steps to reproduce (if any)

@sundb
Copy link
Collaborator

sundb commented Mar 14, 2024

seems related to #12924.
EDIT: not included
@wangbin77 do you have any specific configs?

@wangbin77
Copy link
Author

seems related to #12924. @wangbin77 do you have any specific configs?

No, I only set the auth password

@sundb
Copy link
Collaborator

sundb commented Mar 15, 2024

@wangbin77 ~~Could you share that what size your command is almost? Are there any commands over 32k? ~~
The following memory infos are all overflowing, they are all related to the user, and I'm guessing that one of the user's client structures has been corrupted.
client_recent_max_output_buffer
used_memory_overhead
mem_clients_normal

Could you give the fully crash report? I saw that the client info following Keyspace are not provided. thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants