Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CRASH] Guru Meditation: Unknown btype in replyToBlockedClientTimedOut() #605

Closed
egeneralov opened this issue Mar 14, 2023 · 0 comments · Fixed by #662
Closed

[CRASH] Guru Meditation: Unknown btype in replyToBlockedClientTimedOut() #605

egeneralov opened this issue Mar 14, 2023 · 0 comments · Fixed by #662
Labels
More info needed

Comments

@egeneralov
Copy link

Crash report

Hello, inside container i see next command: keydb-server /etc/keydb/redis.conf --active-replica yes --multi-master yes --appendonly no --bind 0.0.0.0 --port 6379 --protected-mode no --server-threads 2 --masterauth AA000000aaaaaaa --requirepass AA000000aaaaaaa

after connecting clients has an error:

1:1:C 14 Mar 2023 10:18:24.903 * Notice: "active-replica yes" implies "replica-read-only no"
1:1:C 14 Mar 2023 10:18:24.904 # oO0OoO0OoO0Oo KeyDB is starting oO0OoO0OoO0Oo
1:1:C 14 Mar 2023 10:18:24.904 # KeyDB version=6.3.2, bits=64, commit=d1dff8c2, modified=1, pid=1, just started
1:1:C 14 Mar 2023 10:18:24.904 # Configuration loaded
1:1:M 14 Mar 2023 10:18:24.904 * monotonic clock: POSIX clock_gettime
                                                                      
                  _                                                   
               _-(+)-_                                                
            _-- /   \ --_                                            
         _--   /     \   --_            KeyDB  6.3.2 (d1dff8c2/1) 64 bit     
     __--     /       \     --__                                     
    (+) _    /         \    _ (+)       Running in standalone mode
     |   -- /           \ --   |        Port: 6379
     |     /--_   _   _--\     |        PID: 1
     |    /     -(+)-     \    |                                     
     |   /        |        \   |        https://docs.keydb.dev       
     |  /         |         \  |                                     
     | /          |          \ |                                     
    (+)_ -- -- -- | -- -- -- _(+)                                     
        --_       |       _--                                         
            --_   |   _--                                             
                -(+)-        KeyDB has now joined Snap! See the announcement at:  https://docs.keydb.dev/news

                                                                     
1:1:M 14 Mar 2023 10:18:25.193 # Server initialized
1:16:M 14 Mar 2023 10:18:25.194 * Thread 0 alive.
1:17:M 14 Mar 2023 10:18:25.194 * Thread 1 alive.


=== KEYDB BUG REPORT START: Cut & paste starting from here ===
1:16:M 14 Mar 2023 10:19:20.006 # ------------------------------------------------
1:16:M 14 Mar 2023 10:19:20.006 # !!! Software Failure. Press left mouse button to continue
1:16:M 14 Mar 2023 10:19:20.006 # Guru Meditation: Unknown btype in replyToBlockedClientTimedOut(). #blocked.cpp:246

------ STACK TRACE ------

Backtrace:
keydb-server 0.0.0.0:6379(replyToBlockedClientTimedOut(client*)+0x72) [0x5561e31e35b2]
keydb-server 0.0.0.0:6379(checkBlockedClientTimeout(client*, long long)+0x49) [0x5561e31ee5b9]
keydb-server 0.0.0.0:6379(handleBlockedClientsTimeout()+0x127) [0x5561e31ee6f7]
keydb-server 0.0.0.0:6379(beforeSleep(aeEventLoop*)+0x165) [0x5561e3147335]
keydb-server 0.0.0.0:6379(aeProcessEvents+0x10b) [0x5561e3141b0b]
keydb-server 0.0.0.0:6379(aeMain+0x47) [0x5561e31484e7]
keydb-server 0.0.0.0:6379(workerThreadMain(void*)+0x73) [0x5561e32165b3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f046da0d6db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f046d73661f]

------ INFO OUTPUT ------
# Server
redis_version:6.3.2
redis_git_sha1:d1dff8c2
redis_git_dirty:1
redis_build_id:60d46d3de2ba6e96
redis_mode:standalone
os:Linux 5.10.0-20-amd64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:7.5.0
process_id:1
process_supervised:no
run_id:3d63993a80a3062526b120bbef752e3065bf5fba
tcp_port:6379
server_time_usec:1678789160011088
uptime_in_seconds:56
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:1067559
executable:/data/keydb-server
config_file:/etc/keydb/redis.conf

# Clients
connected_clients:8
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:56
client_recent_max_output_buffer:0
blocked_clients:5
tracking_clients:0
clients_in_timeout_table:5
current_client_thread:0
thread_0_clients:8
thread_1_clients:0

# Memory
used_memory:2447184
used_memory_human:2.33M
used_memory_rss:24010752
used_memory_rss_human:22.90M
used_memory_peak:2534360
used_memory_peak_human:2.42M
used_memory_peak_perc:96.56%
used_memory_overhead:2376138
used_memory_startup:2199232
used_memory_dataset:71046
used_memory_dataset_perc:28.65%
allocator_allocated:2858136
allocator_active:3395584
allocator_resident:7131136
total_system_memory:16785612800
total_system_memory_human:15.63G
used_memory_lua:43008
used_memory_lua_human:42.00K
used_memory_scripts:360
used_memory_scripts_human:360B
number_of_cached_scripts:1
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.19
allocator_frag_bytes:537448
allocator_rss_ratio:2.10
allocator_rss_bytes:3735552
rss_overhead_ratio:3.37
rss_overhead_bytes:16879616
mem_fragmentation_ratio:10.13
mem_fragmentation_bytes:21640728
mem_not_counted_for_evict:1048576
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:164186
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
storage_provider:none

# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:1
rdb_bgsave_in_progress:0
rdb_last_save_time:1678789104
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:23
total_commands_processed:91
instantaneous_ops_per_sec:0
total_net_input_bytes:3996
total_net_output_bytes:484
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:9
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:1
dump_payload_sanitizations:0
total_reads_processed:104
total_writes_processed:84
instantaneous_lock_contention:2
avg_lock_contention:0.265625
storage_provider_read_hits:0
storage_provider_read_misses:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:778b111632ae9c2eeee64be3b358aadcde4b44db
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.150568
used_cpu_user:0.230824
used_cpu_sys_children:0.002011
used_cpu_user_children:0.000000
server_threads:2
long_lock_waits:0
used_cpu_sys_main_thread:0.065866
used_cpu_user_main_thread:0.131732

# Modules

# Commandstats
cmdstat_get:calls=9,usec=65,usec_per_call=7.22,rejected_calls=0,failed_calls=0
cmdstat_set:calls=1,usec=50,usec_per_call=50.00,rejected_calls=0,failed_calls=0
cmdstat_brpop:calls=44,usec=538,usec_per_call=12.23,rejected_calls=0,failed_calls=0
cmdstat_eval:calls=1,usec=259,usec_per_call=259.00,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=11,usec=15,usec_per_call=1.36,rejected_calls=0,failed_calls=0
cmdstat_keys:calls=1,usec=141,usec_per_call=141.00,rejected_calls=0,failed_calls=0
cmdstat_evalsha:calls=1,usec=75,usec_per_call=75.00,rejected_calls=0,failed_calls=1
cmdstat_auth:calls=23,usec=174,usec_per_call=7.57,rejected_calls=0,failed_calls=0

# Errorstats
errorstat_NOSCRIPT:count=1

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=1,expires=1,avg_ttl=19994,cached_keys=1

# KeyDB
mvcc_depth:1

------ CLIENT LIST OUTPUT ------
id=13 addr=10.233.69.27:40700 laddr=10.233.67.110:6379 fd=24 name= age=30 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=19 obl=0 oll=0 omem=0 tot-mem=61475 events=r cmd=keys user=default redir=-1
id=14 addr=10.233.69.27:40710 laddr=10.233.67.110:6379 fd=25 name= age=30 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=brpop user=default redir=-1
id=10 addr=10.233.69.27:40692 laddr=10.233.67.110:6379 fd=21 name= age=33 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=21 obl=0 oll=0 omem=0 tot-mem=20533 events=r cmd=brpop user=default redir=-1
id=11 addr=10.233.67.28:41004 laddr=10.233.67.110:6379 fd=22 name= age=30 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=brpop user=default redir=-1
id=12 addr=10.233.67.28:41012 laddr=10.233.67.110:6379 fd=23 name= age=30 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=brpop user=default redir=-1
id=7 addr=10.233.67.28:40980 laddr=10.233.67.110:6379 fd=18 name= age=33 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=21 obl=0 oll=0 omem=0 tot-mem=20533 events=r cmd=brpop user=default redir=-1
id=8 addr=10.233.67.28:40994 laddr=10.233.67.110:6379 fd=19 name= age=33 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=24 obl=0 oll=0 omem=0 tot-mem=20536 events=r cmd=brpop user=default redir=-1
id=9 addr=10.233.69.27:40690 laddr=10.233.67.110:6379 fd=20 name= age=33 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=24 obl=0 oll=0 omem=0 tot-mem=20536 events=r cmd=brpop user=default redir=-1

------ MODULES INFO OUTPUT ------

------ FAST MEMORY TEST ------
1:16:M 14 Mar 2023 10:19:20.014 # main thread terminated
1:16:M 14 Mar 2023 10:19:20.014 # Bio thread for job type #0 terminated
1:16:M 14 Mar 2023 10:19:20.014 # Bio thread for job type #1 terminated
1:16:M 14 Mar 2023 10:19:20.014 # Bio thread for job type #2 terminated

Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

=== KEYDB BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash by opening an issue on github:

           https://github.com/JohnSully/KeyDB/issues

  Suspect RAM error? Use keydb-server --test-memory to verify it.

Aditional information

chart keydb-0.46.1

app version 6.3.2

additional values:

fullnameOverride: pplt-keydb
nodes: 1
password: AA000000aaaaaaa
podDisruptionBudget:
  enabled: false
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
More info needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

keys after brpop causes crash due to timeout not being reset Snapchat/KeyDB
2 participants
@egeneralov @msotheeswaran-sc