[CRASH]

[ghost]

Crash report

Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.

1:24:M 02 Jan 2023 04:42:24.096 * 10 changes in 300 seconds. Saving...
1:24:M 02 Jan 2023 04:42:24.096 * Background saving started
1:46:M 02 Jan 2023 04:42:24.103 * DB saved on disk
1:46:M 02 Jan 2023 04:42:24.105 * RDB: 21 MB of memory used by copy-on-write
1:24:M 02 Jan 2023 04:42:24.197 * Background saving terminated with success


=== KEYDB BUG REPORT START: Cut & paste starting from here ===
1:24:M 02 Jan 2023 04:46:40.240 # ------------------------------------------------
1:24:M 02 Jan 2023 04:46:40.240 # !!! Software Failure. Press left mouse button to continue
1:24:M 02 Jan 2023 04:46:40.240 # Guru Meditation: Unknown btype in replyToBlockedClientTimedOut(). #blocked.cpp:246

------ STACK TRACE ------

Backtrace:
keydb-server *:6379(replyToBlockedClientTimedOut(client*)+0x72) [0x5591b9872552]
keydb-server *:6379(checkBlockedClientTimeout(client*, long long)+0x49) [0x5591b992eb99]
keydb-server *:6379(handleBlockedClientsTimeout()+0x127) [0x5591b992ecd7]
keydb-server *:6379(beforeSleep(aeEventLoop*)+0x155) [0x5591b9866d35]
keydb-server *:6379(aeProcessEvents+0x10b) [0x5591b985ea1b]
keydb-server *:6379(aeMain+0x47) [0x5591b9868467]
keydb-server *:6379(workerThreadMain(void*)+0x73) [0x5591b99384b3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fdc792ac6db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fdc78fd561f]

------ INFO OUTPUT ------
# Server
redis_version:6.3.1
redis_git_sha1:ee16abf0
redis_git_dirty:1
redis_build_id:46da54e1e1000581
redis_mode:standalone
os:Linux 6.0.9-arch1-1 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:7.5.0
process_id:1
process_supervised:no
run_id:f0c8c2f6c105d5520f28be372d186b81d579f53c
tcp_port:6379
server_time_usec:1672634800242286
uptime_in_seconds:139008
uptime_in_days:1
hz:10
configured_hz:10
lru_clock:11690416
executable:/data/keydb-server
config_file:/etc/keydb/keydb.conf

# Clients
connected_clients:2
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:32
client_recent_max_output_buffer:0
blocked_clients:1
tracking_clients:0
clients_in_timeout_table:1
current_client_thread:0
thread_0_clients:2
thread_1_clients:0

# Memory
used_memory:2119752
used_memory_human:2.02M
used_memory_rss:27824128
used_memory_rss_human:26.54M
used_memory_peak:278921568
used_memory_peak_human:266.00M
used_memory_peak_perc:0.76%
used_memory_overhead:2038824
used_memory_startup:1989496
used_memory_dataset:80928
used_memory_dataset_perc:62.13%
allocator_allocated:2750672
allocator_active:5664768
allocator_resident:13594624
total_system_memory:33577005056
total_system_memory_human:31.27G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:2.06
allocator_frag_bytes:2914096
allocator_rss_ratio:2.40
allocator_rss_bytes:7929856
rss_overhead_ratio:2.05
rss_overhead_bytes:14229504
mem_fragmentation_ratio:13.62
mem_fragmentation_bytes:25780704
mem_not_counted_for_evict:1048576
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:41008
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
storage_provider:none

# Persistence
loading:0
current_cow_size:22478848
current_cow_size_age:256
current_fork_perc:0.00
current_save_keys_processed:1
current_save_keys_total:0
rdb_changes_since_last_save:44
rdb_bgsave_in_progress:0
rdb_last_save_time:1672634544
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:22478848
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:40
total_commands_processed:124635
instantaneous_ops_per_sec:0
total_net_input_bytes:134988398
total_net_output_bytes:975706
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:1026
keyspace_misses:10
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:21
dump_payload_sanitizations:0
total_reads_processed:81319
total_writes_processed:80457
instantaneous_lock_contention:2
avg_lock_contention:0.000000
storage_provider_read_hits:0
storage_provider_read_misses:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:c20878f308c580b64eb78ad342b06d5256e7eb0c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:162.387607
used_cpu_user:366.253957
used_cpu_sys_children:0.001478
used_cpu_user_children:0.001538
server_threads:2
long_lock_waits:8
used_cpu_sys_main_thread:94.482873
used_cpu_user_main_thread:210.632029

# Modules

# Commandstats
cmdstat_unlink:calls=30378,usec=28271,usec_per_call=0.93,rejected_calls=0,failed_calls=0
cmdstat_command:calls=1,usec=839,usec_per_call=839.00,rejected_calls=0,failed_calls=0
cmdstat_blpop:calls=25,usec=299,usec_per_call=11.96,rejected_calls=0,failed_calls=0
cmdstat_hmset:calls=92643,usec=2952514,usec_per_call=31.87,rejected_calls=0,failed_calls=0
cmdstat_config:calls=30,usec=335,usec_per_call=11.17,rejected_calls=0,failed_calls=0
cmdstat_rpush:calls=334,usec=2869,usec_per_call=8.59,rejected_calls=0,failed_calls=0
cmdstat_flushdb:calls=1,usec=131936,usec_per_call=131936.00,rejected_calls=0,failed_calls=0
cmdstat_hgetall:calls=21,usec=332,usec_per_call=15.81,rejected_calls=0,failed_calls=0
cmdstat_client:calls=20,usec=48,usec_per_call=2.40,rejected_calls=0,failed_calls=0
cmdstat_select:calls=19,usec=28,usec_per_call=1.47,rejected_calls=0,failed_calls=0
cmdstat_info:calls=76,usec=6130,usec_per_call=80.66,rejected_calls=0,failed_calls=0
cmdstat_hdel:calls=3,usec=4,usec_per_call=1.33,rejected_calls=0,failed_calls=0
cmdstat_get:calls=10,usec=27,usec_per_call=2.70,rejected_calls=0,failed_calls=0
cmdstat_hset:calls=16,usec=112,usec_per_call=7.00,rejected_calls=0,failed_calls=0
cmdstat_cluster:calls=10,usec=7,usec_per_call=0.70,rejected_calls=0,failed_calls=10
cmdstat_echo:calls=20,usec=8,usec_per_call=0.40,rejected_calls=0,failed_calls=0
cmdstat_subscribe:calls=10,usec=33,usec_per_call=3.30,rejected_calls=0,failed_calls=0
cmdstat_del:calls=12,usec=25,usec_per_call=2.08,rejected_calls=0,failed_calls=0
cmdstat_hget:calls=1005,usec=1218,usec_per_call=1.21,rejected_calls=0,failed_calls=0
cmdstat_keys:calls=1,usec=30,usec_per_call=30.00,rejected_calls=0,failed_calls=0

# Errorstats
errorstat_ERR:count=21

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=3,expires=0,avg_ttl=0,cached_keys=3

# KeyDB
mvcc_depth:1

------ CLIENT LIST OUTPUT ------
id=17 addr=127.0.0.1:35486 laddr=127.0.0.1:6379 fd=20 name= age=571 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=6 obl=0 oll=0 omem=0 tot-mem=61462 events=r cmd=keys user=default redir=-1
id=43 addr=172.17.0.1:52866 laddr=172.17.0.5:6379 fd=31 name= age=13 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=info user=default redir=-1

------ MODULES INFO OUTPUT ------

------ FAST MEMORY TEST ------
1:24:M 02 Jan 2023 04:46:40.242 # main thread terminated
1:24:M 02 Jan 2023 04:46:40.242 # Bio thread for job type #0 terminated
1:24:M 02 Jan 2023 04:46:40.242 # Bio thread for job type #1 terminated
1:24:M 02 Jan 2023 04:46:40.242 # Bio thread for job type #2 terminated

Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

=== KEYDB BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash by opening an issue on github:

           https://github.com/JohnSully/KeyDB/issues

  Suspect RAM error? Use keydb-server --test-memory to verify it.

Aditional information

1. OS distribution and version
   Docker eqalpha/keydb a7d31975720e, on Arch Linux host

2. Steps to reproduce (if any)
   I don't know how to reproduce this.

[msotheeswaran-sc]

Even if you are not sure how to reproduce this, context on the configs you are using and the commands that were running can help us investigate.

[ghost]

Sure, no problem. I was running the default configuration: docker run --name redis -p localhost:6379:6379 eqalpha/keydb, and was doing HSET commands, ~4k/sec.

[kklash]

Hi there, I have also encountered this crash. I was running KeyDB as a systemd service.

Logs, config file and systemd unit file are attached: keydb-debugging.zip