Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed assertion: <no assertion failed> (<no file>:0) #303

Closed
2peter3 opened this issue Apr 1, 2021 · 3 comments
Closed

Failed assertion: <no assertion failed> (<no file>:0) #303

2peter3 opened this issue Apr 1, 2021 · 3 comments
Labels
Priority 1

Comments

@2peter3
Copy link

2peter3 commented Apr 1, 2021

Describe the bug
After the Docker Container with the latest stable/unstable Version crashed and i try to restart the KeyDB, im getting
this infinity loop of errors.

** Log Files **

1:1:C 01 Apr 2021 17:32:28.406 # oO0OoO0OoO0Oo KeyDB is starting oO0OoO0OoO0Oo
 1:1:C 01 Apr 2021 17:32:28.406 # KeyDB version=255.255.255, bits=64, commit=00000000, modified=0, pid=1, just started
 1:1:C 01 Apr 2021 17:32:28.406 # Configuration loaded
 
                   _
                _-(+)-_
             _-- /   \ --_
          _--   /     \   --_            KeyDB 255.255.255 (00000000/0) 64 bit
      __--     /       \     --__
     (+) _    /         \    _ (+)       Running in standalone mode
      |   -- /           \ --   |        Port: 6379
      |     /--_   _   _--\     |        PID: 1
      |    /     -(+)-     \    |
      |   /        |        \   |        https://docs.keydb.dev
      |  /         |         \  |
      | /          |          \ |
     (+)_ -- -- -- | -- -- -- _(+)
         --_       |       _--
             --_   |   _--
                 -(+)-        Join the KeyDB community! https://community.keydb.dev/
 
 
 1:1:M 01 Apr 2021 17:32:28.662 # WARNING: The TCP backlog setting of 2048 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
 1:1:M 01 Apr 2021 17:32:28.662 # Server initialized
 1:1:M 01 Apr 2021 17:32:28.663 * Loading RDB produced by version 255.255.255
 1:1:M 01 Apr 2021 17:32:28.663 * RDB age 522 seconds
 1:1:M 01 Apr 2021 17:32:28.663 * RDB memory usage when created 731.18 Mb
 1:1:M 01 Apr 2021 17:32:38.055 * DB loaded from disk: 9.392 seconds
 1:35:M 01 Apr 2021 17:32:38.055   Thread 0 alive.
 1:35:M 01 Apr 2021 17:33:30.600 # Lua slow script detected: still in execution after 11699 milliseconds. You can try killing the script using the SCRIPT KILL command. Script SHA1 is: aa102f22e4bd254581ad75f64cb52ea1cc1aa128
 
 
 === KEYDB BUG REPORT START: Cut & paste starting from here ===
 1:35:M 01 Apr 2021 17:33:41.932 # KeyDB 255.255.255 crashed by signal: 11, si_code: 1
 1:35:M 01 Apr 2021 17:33:41.932 # Crashed running the instruction at: 0x55dbb0c2ac58
 1:35:M 01 Apr 2021 17:33:41.932 # Accessing address: 0xffffffffffffffff
 1:35:M 01 Apr 2021 17:33:41.932 # Failed assertion: <no assertion failed> (<no file>:0)
 
 ------ STACK TRACE ------
 EIP:
 keydb-server *:6379(processInputBuffer(client*, int)+0x98) [0x55dbb0c2ac58]
 
 Backtrace:
 keydb-server *:6379(logStackTrace(ucontext_t*)+0x6b) [0x55dbb0c71ecb]
 keydb-server *:6379(sigsegvHandler(int, siginfo_t*, void*)+0xed) [0x55dbb0c726ad]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f6ae4282980]
 keydb-server *:6379(processInputBuffer(client*, int)+0x98) [0x55dbb0c2ac58]
 keydb-server *:6379(processClients()+0xae) [0x55dbb0c2da9e]
 keydb-server *:6379(beforeSleep(aeEventLoop*)+0xe4) [0x55dbb0c10424]
 keydb-server *:6379(aeMain+0x89) [0x55dbb0c0c5a9]
 keydb-server *:6379(workerThreadMain(void*)+0x7a) [0x55dbb0c1386a]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f6ae42776db]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f6ae3fa071f]
 
 ------ INFO OUTPUT ------
 # Server
 redis_version:255.255.255
 redis_git_sha1:00000000
 redis_git_dirty:0
 redis_build_id:f5c4e4b31d2d686d
 redis_mode:standalone
 os:Linux 4.19.0-16-amd64 x86_64
 arch_bits:64
 multiplexing_api:epoll
 atomicvar_api:atomic-builtin
 gcc_version:7.5.0
 process_id:1
 run_id:67478145b164db14d666e61cec3e2780eb632a9e
 tcp_port:6379
 uptime_in_seconds:73
 uptime_in_days:0
 hz:10
 configured_hz:10
 lru_clock:6685653
 executable:/data/keydb-server
 config_file:/etc/keydb/keydb.conf
 
 # Clients
 connected_clients:5
 client_recent_max_input_buffer:1024
 client_recent_max_output_buffer:0
 blocked_clients:0
 tracking_clients:0
 clients_in_timeout_table:0
 current_client_thread:0
 thread_0_clients:5
 
 # Memory
 used_memory:688436128
 used_memory_human:656.54M
 used_memory_rss:719921152
 used_memory_rss_human:686.57M
 used_memory_peak:702588744
 used_memory_peak_human:670.04M
 used_memory_peak_perc:97.99%
 used_memory_overhead:6339680
 used_memory_startup:6228376
 used_memory_dataset:682096448
 used_memory_dataset_perc:99.98%
 allocator_allocated:700005488
 allocator_active:703070208
 allocator_resident:719642624
 total_system_memory:16820826112
 total_system_memory_human:15.67G
 used_memory_lua:3099648
 used_memory_lua_human:2.96M
 used_memory_scripts:5800
 used_memory_scripts_human:5.66K
 number_of_cached_scripts:5
 maxmemory:0
 maxmemory_human:0B
 maxmemory_policy:noeviction
 allocator_frag_ratio:1.00
 allocator_frag_bytes:3064720
 allocator_rss_ratio:1.02
 allocator_rss_bytes:16572416
 rss_overhead_ratio:1.00
 rss_overhead_bytes:278528
 mem_fragmentation_ratio:1.03
 mem_fragmentation_bytes:20010728
 mem_not_counted_for_evict:0
 mem_replication_backlog:0
 mem_clients_slaves:0
 mem_clients_normal:103152
 mem_aof_buffer:0
 mem_allocator:jemalloc-5.2.1
 active_defrag_running:0
 lazyfree_pending_objects:0
 
 # Persistence
 loading:0
 rdb_changes_since_last_save:166730
 rdb_bgsave_in_progress:0
 rdb_last_save_time:1617298348
 rdb_last_bgsave_status:ok
 rdb_last_bgsave_time_sec:-1
 rdb_current_bgsave_time_sec:-1
 rdb_last_cow_size:0
 aof_enabled:0
 aof_rewrite_in_progress:0
 aof_rewrite_scheduled:0
 aof_last_rewrite_time_sec:-1
 aof_current_rewrite_time_sec:-1
 aof_last_bgrewrite_status:ok
 aof_last_write_status:ok
 aof_last_cow_size:0
 module_fork_in_progress:0
 module_fork_last_cow_size:0
 
 # Stats
 total_connections_received:241
 total_commands_processed:226193
 instantaneous_ops_per_sec:5063
 total_net_input_bytes:12467194
 total_net_output_bytes:58646747
 instantaneous_input_kbps:262.94
 instantaneous_output_kbps:1525.80
 rejected_connections:0
 sync_full:0
 sync_partial_ok:0
 sync_partial_err:0
 expired_keys:2
 expired_stale_perc:0.00
 expired_time_cap_reached_count:0
 expire_cycle_cpu_milliseconds:0
 evicted_keys:0
 keyspace_hits:56034
 keyspace_misses:181
 pubsub_channels:1
 pubsub_patterns:0
 latest_fork_usec:0
 migrate_cached_sockets:0
 slave_expires_tracked_keys:0
 active_defrag_hits:0
 active_defrag_misses:0
 active_defrag_key_hits:0
 active_defrag_key_misses:0
 tracking_total_keys:0
 tracking_total_items:0
 tracking_total_prefixes:0
 unexpected_error_replies:0
 total_reads_processed:67959
 total_writes_processed:225760
 
 # Replication
 role:master
 connected_slaves:0
 master_replid:f6d00f42030e6333b791524913a1efe47ec8bf52
 master_replid2:0000000000000000000000000000000000000000
 master_repl_offset:0
 second_repl_offset:-1
 repl_backlog_active:0
 repl_backlog_size:1048576
 repl_backlog_first_byte_offset:0
 repl_backlog_histlen:0
 
 # CPU
 used_cpu_sys:13.339255
 used_cpu_user:55.249342
 used_cpu_sys_children:0.004308
 used_cpu_user_children:0.003735
 server_threads:1
 long_lock_waits:0
 
 # Modules
 
 # Commandstats
 cmdstat_del:calls=3,usec=4895,usec_per_call=1631.67
 cmdstat_hset:calls=9679,usec=31585,usec_per_call=3.26
 cmdstat_multi:calls=10301,usec=4780,usec_per_call=0.46
 cmdstat_scard:calls=120,usec=40,usec_per_call=0.33
 cmdstat_hexists:calls=2885,usec=6531,usec_per_call=2.26
 cmdstat_keys:calls=1,usec=17,usec_per_call=17.00
 cmdstat_command:calls=1,usec=2101,usec_per_call=2101.00
 cmdstat_exec:calls=7645,usec=15415187,usec_per_call=2016.38
 cmdstat_srem:calls=41928,usec=75860,usec_per_call=1.81
 cmdstat_zcard:calls=60,usec=78,usec_per_call=1.30
 cmdstat_script:calls=310,usec=947,usec_per_call=3.05
 cmdstat_llen:calls=120,usec=118,usec_per_call=0.98
 cmdstat_hget:calls=52957,usec=124898,usec_per_call=2.36
 cmdstat_publish:calls=7585,usec=476338,usec_per_call=62.80
 cmdstat_incr:calls=2885,usec=5946,usec_per_call=2.06
 cmdstat_evalsha:calls=3067,usec=33996681,usec_per_call=11084.67
 cmdstat_brpoplpush:calls=32959,usec=185696,usec_per_call=5.63
 cmdstat_info:calls=2182,usec=85471,usec_per_call=39.17
 cmdstat_set:calls=182,usec=316,usec_per_call=1.74
 cmdstat_subscribe:calls=60,usec=176,usec_per_call=2.93
 cmdstat_lrem:calls=40476,usec=48297207,usec_per_call=1193.23
 cmdstat_hdel:calls=7585,usec=28626,usec_per_call=3.77
 cmdstat_get:calls=60,usec=32,usec_per_call=0.53
 cmdstat_lrange:calls=3,usec=4033,usec_per_call=1344.33
 cmdstat_lpush:calls=2919,usec=24344,usec_per_call=8.34
 cmdstat_hlen:calls=7,usec=9,usec_per_call=1.29
 cmdstat_sadd:calls=30,usec=8180,usec_per_call=272.67
 cmdstat_smembers:calls=3,usec=6095,usec_per_call=2031.67
 cmdstat_select:calls=180,usec=109,usec_per_call=0.61
 
 # Cluster
 cluster_enabled:0
 
 # Keyspace
 db0:keys=5,expires=1,avg_ttl=0
 
 ------ CLIENT LIST OUTPUT ------
 id=207 addr=10.0.14.6:54568 fd=217 name= age=60 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=hget user=default
 id=184 addr=127.0.0.1:60216 fd=194 name= age=61 idle=49 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=hlen user=default
 id=162 addr=10.0.14.6:47954 fd=173 name= age=67 idle=32 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default
 id=163 addr=10.0.14.6:47956 fd=174 name= age=67 idle=32 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 argv-mem=0 obl=0 oll=17 omem=328112 tot-mem=389616 events=r cmd=evalsha user=default
 id=164 addr=10.0.14.6:47958 fd=175 name= age=67 idle=32 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=brpoplpush user=default
 
 ------ REGISTERS ------
 1:35:M 01 Apr 2021 17:33:41.933 #
 RAX:000055dbb0fe1e20 RBX:00007f6aa6785c40
 RCX:0000000000000000 RDX:000055dbb0fe1e48
 RDI:0000000000000000 RSI:00007f6ade42f100
 RBP:ffffffffffffffe0 RSP:00007f6aa7ffd4a0
 R8 :0000000000000000 R9 :0000000000001000
 R10:0000000000000001 R11:0000000000000010
 R12:000000000000000f R13:000055dbb0d74f0c
 R14:0000000000000000 R15:00007ffccde0c2c0
 RIP:000055dbb0c2ac58 EFL:0000000000010246
 CSGSFS:002b000000000033
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4af) -> 000055dbb0c10424
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4ae) -> 0000000000000000
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4ad) -> ffffffffffffffff
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4ac) -> 00007f6ade42f100
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4ab) -> 0000000000000000
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4aa) -> 000055dbb0fe1b20
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4a9) -> 4dc657ff53727300
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4a8) -> 0000000000000000
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4a7) -> 000055dbb0c0c101
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4a6) -> 00007f6aa6789edc
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4a5) -> 000055dbb0c2da9e
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4a4) -> 0000000000000000
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4a3) -> ffffffffffffffe0
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4a2) -> 00007f6aa72051d8
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4a1) -> 00007f6aa6785c40
 1:35:M 01 Apr 2021 17:33:41.933 # (00007f6aa7ffd4a0) -> 00007f6aa7205058
 
 ------ MODULES INFO OUTPUT ------
 
 ------ FAST MEMORY TEST ------
 1:35:M 01 Apr 2021 17:33:41.934 # main thread terminated
 1:35:M 01 Apr 2021 17:33:41.934 # Bio thread for job type #0 terminated
 1:35:M 01 Apr 2021 17:33:41.934 # Bio thread for job type #1 terminated
 1:35:M 01 Apr 2021 17:33:41.934 # Bio thread for job type #2 terminated
 
 Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
 
 ------ DUMPING CODE AROUND EIP ------
 Symbol: _Z18processInputBufferP6clienti (base: 0x55dbb0c2abc0)
 Module: keydb-server *:6379 (base 0x55dbb0bbe000)
 $ xxd -r -p /tmp/dump.hex /tmp/dump.bin
 $ objdump --adjust-vma=0x55dbb0c2abc0 -D -b binary -m i386:x86-64 /tmp/dump.bin
 ------
 1:35:M 01 Apr 2021 17:33:41.934 # dump of function (hexdump of 280 bytes):
 415641554154554189f4488b2d6fd33a00534889fb8b8f9842000064488b55004885d2741f4863c1488d3480488d04f0488b3248c1e004480305d2d73a0048397028743d85c9743048837b08007432488d3d4e451400488d35b79c1400ba04090000e8c9590400bf01000000e82f87fdff0f1f8000000000e8b30d080085c074c74c8d2dc4a214000f1f840000000000488b7b28488b4b300fb647ff4889ce89c283e2070f8556010000c1f80348984839c10f83d8000000488b83c00000004c8db3c0000000a801751364488b45008bb09000000085f60f853b010000488b83c0000000a8100f852c010000488b83c0000000a9000000400f851a010000488b050bd73a008b90ec35000085d2740f488b83c0000000a802
 Function at 0x55dbb0c705f0 is _serverAssert
 Function at 0x55dbb0cab9f0 is _Z25moduleGILAcquiredByModulev
 
 === KEYDB BUG REPORT END. Make sure to include from START to END. ===
 
        Please report the crash by opening an issue on github:
 
            https://github.com/JohnSully/KeyDB/issues
 
   Suspect RAM error? Use keydb-server --test-memory to verify it.

To Reproduce
no sorry :/
@JohnSully
Copy link
Collaborator

Notes: Crash after slow lua script, perhaps processClientsWhileBlocked related again.

@smarakdas314
Copy link

Not sure I would need to open a new issue, or I could just paste here:

After upgrading from 6.2.x to 6.3.x started getting some "Assertion Failed" events like so:

=== KEYDB BUG REPORT START: Cut & paste starting from here ===
1392970:1393109:M 19 Aug 2022 10:59:21.369 # === ASSERTION FAILED ===
1392970:1393109:M 19 Aug 2022 10:59:21.369 # ==> dict.cpp:986 'iter->fingerprint == dictFingerprint(iter->d)' is not true

------ STACK TRACE ------

Backtrace:
/usr/bin/keydb-server *:6379(+0x174578) [0x55aa170b3578]
/usr/bin/keydb-server *:6379(rdbSaveObject(_rio*, robj_roptr, robj_roptr)+0x598) [0x55aa1704da18]
/usr/bin/keydb-server *:6379(rdbSaveKeyValuePair(_rio*, robj_roptr, robj_roptr, expireEntry const*)+0x108) [0x55aa1704e028]
/usr/bin/keydb-server *:6379(saveKey(_rio*, redisDbPersistentDataSnapshot const*, int, unsigned long*, char const*, robj_roptr)+0x85) [0x55aa170517c5]
/usr/bin/keydb-server *:6379(+0x10385b) [0x55aa1704285b]
/usr/bin/keydb-server *:6379(redisDbPersistentDataSnapshot::iterate_threadsafe_core(std::function<bool (char const*, robj_roptr)>&, bool, bool, bool) const+0xa6) [0x55aa16fa3006]
/usr/bin/keydb-server *:6379(rdbSaveRio(_rio*, redisDbPersistentDataSnapshot const**, int*, int, rdbSaveInfo*)+0x285) [0x55aa1704bde5]
/usr/bin/keydb-server *:6379(rdbSaveFile(char*, redisDbPersistentDataSnapshot const**, rdbSaveInfo*)+0xc9) [0x55aa1704c179]
/usr/bin/keydb-server *:6379(rdbSave(redisDbPersistentDataSnapshot const**, rdbSaveInfo*)+0x6a) [0x55aa1704c4ca]
/usr/bin/keydb-server *:6379(rdbSaveThread(void*)+0x24e) [0x55aa1705360e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f7d07bdafa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f7d07b0b4cf]

------ INFO OUTPUT ------
# Server
redis_version:6.3.1

@msotheeswaran-sc
Copy link
Collaborator

Should be fixed with 6.3.2, please re-open if you run into this again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Priority 1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@JohnSully @2peter3 @smarakdas314 @msotheeswaran-sc