Deadlock in replicationFeedMonitors #150
Comments
|
This should be fixed in v5.3.2. The line item in the release note: 'Fix crash propogating stale keys in active replication' is the same issue as you have here. Effectively the expired key is not visible but still exists in the database, causing this assert to fire when we try to overwrite it. If 5.3.2 doesn't resolve the problem let me know. |
|
Thanks for the info, |
|
5.3.2 is already out. Your crash report shows you are on 5.3.0. |
|
Tried same on 5.3.2, this time with different result: 16350:M 18 Mar 2020 11:26:08.028 # !!! ERROR: Deadlock detected !!! === KEYDB BUG REPORT START: Cut & paste starting from here === ------ STACK TRACE ------ Backtrace: |
JohnSully
changed the title
|
That is a new issue. I’ve updated the title. |
|
I believe this is fixed with change: 3c37349 I have also updated the deadlock detector to print stack traces for all affected threads. This will make it possible to better pinpoint these sorts of issues in the future. |
|
The proposed fix is released in v5.3.3. Please let me know if this resolves your issue. Should you still have problems the new version will print out callstacks for each thread involved in the deadlock, this extra logging information is important for tracking these down so please include it in any updates. |
|
2 months with no additional reports, closing. |
Hi John, Just a quick update regarding the issue in v5.3.3, I tested this and seems like the issue is gone. Thanks! |
|
Hi @mcrivar I did find a separate hang that is fixed in 6.0.8. I would recommend going with that build to ensure there are no further issues. |
Describe the bug
I'm running keydb 5.3.0 in ubuntu18 with master-slave configuration where the master is redis and the slave is keydb.
I'm trying to sync DB from master with ~30gb size.
When the sync is finished and the slave(keydb) starts to load DB in memory in some point it crashes and fails.
When loading the aof file locally on the keydb host without master-slave configuration everything runs ok and keydb starts.
Please advice.
** Log Files **
These should be KeyDB logs, not syslogs or logs from your container manager. If you are reporting a crash there will be a line in your log stating:
=== KEYDB BUG REPORT START: Cut & paste starting from here ===
9308:S 05 Mar 2020 12:48:14.844 # === ASSERTION FAILED OBJECT CONTEXT ===
9308:S 05 Mar 2020 12:48:14.844 # Object type: 0
9308:S 05 Mar 2020 12:48:14.844 # Object encoding: 0
9308:S 05 Mar 2020 12:48:14.844 # Object refcount: 1
9308:S 05 Mar 2020 12:48:14.844 # Object raw string len: 52
9308:S 05 Mar 2020 12:48:14.844 # Object raw string content: "long47133_157122741542325b2f70e258f0832e123c0c815427"
9308:S 05 Mar 2020 12:48:14.844 # === ASSERTION FAILED ===
9308:S 05 Mar 2020 12:48:14.844 # ==> db.cpp:226 'retval == DICT_OK' is not true
9308:S 05 Mar 2020 12:48:14.844 # (forcing SIGSEGV to print the bug report.)
9308:S 05 Mar 2020 12:48:14.844 # KeyDB 5.3.0 crashed by signal: 11
9308:S 05 Mar 2020 12:48:14.844 # Crashed running the instruction at: 0x55c80e3be2ce
9308:S 05 Mar 2020 12:48:14.844 # Accessing address: 0xffffffffffffffff
9308:S 05 Mar 2020 12:48:14.844 # Failed assertion: retval == DICT_OK (db.cpp:226)
------ STACK TRACE ------
EIP:
/usr/local/bin/keydb-server 0.0.0.0:6767(_serverAssert+0x7e) [0x55c80e3be2ce]
Backtrace:
/usr/local/bin/keydb-server 0.0.0.0:6767(logStackTrace(ucontext_t*)+0x6b) [0x55c80e3bfa5b]
/usr/local/bin/keydb-server 0.0.0.0:6767(sigsegvHandler(int, siginfo_t*, void*)+0xc9) [0x55c80e3c01f9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f3d7906e890]
/usr/local/bin/keydb-server 0.0.0.0:6767(_serverAssert+0x7e) [0x55c80e3be2ce]
/usr/local/bin/keydb-server 0.0.0.0:6767(dbAdd(redisDb*, redisObject*, redisObject*)+0x2d) [0x55c80e3851fd]
/usr/local/bin/keydb-server 0.0.0.0:6767(incrDecrCommand(client*, long long)+0x19c) [0x55c80e39ec5c]
/usr/local/bin/keydb-server 0.0.0.0:6767(call(client*, int)+0xcd) [0x55c80e367f4d]
/usr/local/bin/keydb-server 0.0.0.0:6767(processCommand(client*, int)+0x7f4) [0x55c80e36c6c4]
/usr/local/bin/keydb-server 0.0.0.0:6767(processCommandAndResetClient(client*, int)+0x21) [0x55c80e3763a1]
/usr/local/bin/keydb-server 0.0.0.0:6767(processInputBuffer(client*, int)+0x18b) [0x55c80e37b11b]
/usr/local/bin/keydb-server 0.0.0.0:6767(processInputBufferAndReplicate(client*)+0x2d) [0x55c80e37b21d]
/usr/local/bin/keydb-server 0.0.0.0:6767(readQueryFromClient(aeEventLoop*, int, void*, int)+0x370) [0x55c80e37dd90]
/usr/local/bin/keydb-server 0.0.0.0:6767(ProcessEventCore+0x11c) [0x55c80e36037c]
/usr/local/bin/keydb-server 0.0.0.0:6767(aeProcessEvents+0x165) [0x55c80e360615]
/usr/local/bin/keydb-server 0.0.0.0:6767(aeMain+0x45) [0x55c80e3609e5]
/usr/local/bin/keydb-server 0.0.0.0:6767(workerThreadMain(void*)+0x9e) [0x55c80e364f0e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f3d790636db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3d78d8c88f]
------ INFO OUTPUT ------
Server
redis_version:5.3.0
redis_git_sha1:d0654af7
redis_git_dirty:0
redis_build_id:a1fad2f51fcb9f0c
redis_mode:standalone
os:Linux 4.15.0-88-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:7.4.0
process_id:9308
run_id:69e64c591048ff4a508c824b1bbb0bbb8f303fe8
tcp_port:6767
uptime_in_seconds:3272
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:6354190
executable:/usr/local/bin/keydb-server
config_file:/etc/keydb/keydb-6767.conf
Clients
connected_clients:1
client_recent_max_input_buffer:4
client_recent_max_output_buffer:0
blocked_clients:0
current_client_thread:0
thread_0_clients:1
thread_1_clients:0
thread_2_clients:0
thread_3_clients:0
thread_4_clients:0
thread_5_clients:0
thread_6_clients:0
thread_7_clients:0
Memory
used_memory:46216288464
used_memory_human:43.04G
used_memory_rss:101767344128
used_memory_rss_human:94.78G
used_memory_peak:46216288464
used_memory_peak_human:43.04G
used_memory_peak_perc:100.00%
used_memory_overhead:27349533352
used_memory_startup:9023848
used_memory_dataset:18866755112
used_memory_dataset_perc:40.83%
allocator_allocated:46216275072
allocator_active:101767306240
allocator_resident:101767306240
total_system_memory:270388236288
total_system_memory_human:251.82G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:805306368000
maxmemory_human:750.00G
maxmemory_policy:noeviction
allocator_frag_ratio:2.20
allocator_frag_bytes:55551031168
allocator_rss_ratio:1.00
allocator_rss_bytes:0
rss_overhead_ratio:1.00
rss_overhead_bytes:37888
mem_fragmentation_ratio:2.20
mem_fragmentation_bytes:55551069056
mem_not_counted_for_evict:321446
mem_replication_backlog:1073745904
mem_clients_slaves:0
mem_clients_normal:49906
mem_aof_buffer:321446
mem_allocator:libc
active_defrag_running:0
lazyfree_pending_objects:0
Persistence
loading:0
rdb_changes_since_last_save:61392667
rdb_bgsave_in_progress:0
rdb_last_save_time:1583409222
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:1
aof_rewrite_in_progress:1
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:0
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
aof_current_size:25640652586
aof_base_size:25413737489
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:321446
aof_pending_bio_fsync:0
aof_delayed_fsync:0
Stats
total_connections_received:12
total_commands_processed:9494
instantaneous_ops_per_sec:3412
total_net_input_bytes:21237370471
total_net_output_bytes:189
instantaneous_input_kbps:203408.98
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:3507136
expired_stale_perc:0.00
expired_time_cap_reached_count:1054
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:1380440
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
Replication
role:slave
Master 0:
master_host:""
master_port:6767
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:27006718967
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:123042c3d498ea87ac21a2931dea8e5458887b95
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:27006706749
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1073741824
repl_backlog_first_byte_offset:27006067777
repl_backlog_histlen:638973
CPU
used_cpu_sys:452.135519
used_cpu_user:2789.435190
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
server_threads:8
long_lock_waits:72697
Commandstats
cmdstat_select:calls=1,usec=1,usec_per_call=1.00
cmdstat_ping:calls=2,usec=3,usec_per_call=1.50
cmdstat_set:calls=18,usec=18,usec_per_call=1.00
cmdstat_slaveof:calls=1,usec=110,usec_per_call=110.00
cmdstat_incr:calls=4892,usec=84287,usec_per_call=17.23
cmdstat_del:calls=2296,usec=53696,usec_per_call=23.39
cmdstat_dbsize:calls=6,usec=11,usec_per_call=1.83
cmdstat_expire:calls=2278,usec=3856,usec_per_call=1.69
Cluster
cluster_enabled:0
Keyspace
db0:keys=365432811,expires=260074014,avg_ttl=284433301
------ CLIENT LIST OUTPUT ------
id=23 addr=?:0 fd=92 name= age=1 idle=0 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=16387 qbuf-free=16385 obl=0 oll=0 omem=0 events=r cmd=incr
------ CURRENT CLIENT INFO ------
id=23 addr=?:0 fd=92 name= age=1 idle=0 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=16387 qbuf-free=16385 obl=0 oll=0 omem=0 events=r cmd=incr
argv[0]: 'incr'
argv[1]: 'long47133_157122741542325b2f70e258f0832e123c0c815427'
9308:S 05 Mar 2020 12:48:14.890 # key 'long47133_157122741542325b2f70e258f0832e123c0c815427' found in DB containing the following object:
9308:S 05 Mar 2020 12:48:14.890 # Object type: 0
9308:S 05 Mar 2020 12:48:14.890 # Object encoding: 1
9308:S 05 Mar 2020 12:48:14.890 # Object refcount: 1
------ REGISTERS ------
9308:S 05 Mar 2020 12:48:14.890 #
RAX:0000000000000000 RBX:00000000000000e2
RCX:0000000000000007 RDX:00007f3c600008d0
RDI:000055d275bcbbd0 RSI:00007f3c600009d8
RBP:000055c80e43c197 RSP:00007f3c76e67b10
R8 :00007f3c76e68700 R9 :00007f3c76e68700
R10:00000000ffffffd6 R11:0000000000000206
R12:000055c80e438e66 R13:000055c80e67ca10
R14:00007f30f2bfab90 R15:0000000000000000
RIP:000055c80e3be2ce EFL:0000000000010202
CSGSFS:002b000000000033
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b1f) -> 00007f3c76e67b90
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b1e) -> 0000000000000002
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b1d) -> 0000000003a8c71b
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b1c) -> 0000000000000018
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b1b) -> 000055c80e367f4d
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b1a) -> 0005a01af349c23e
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b19) -> 000000000000000f
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b18) -> 00007f30f2bfac40
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b17) -> 23e4c564d859b800
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b16) -> 0000000000000001
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b15) -> 000055c80e39ec5c
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b14) -> 00007f30f2bfab90
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b13) -> 000055c80e3851fd
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b12) -> 0000000000000001
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b11) -> 000055c80fcf6a10
9308:S 05 Mar 2020 12:48:14.890 # (00007f3c76e67b10) -> 00007f39ad6e5e00
------ FAST MEMORY TEST ------
9308:S 05 Mar 2020 12:48:14.891 # Bio thread for job type #0 terminated
9308:S 05 Mar 2020 12:48:14.891 # Bio thread for job type #1 terminated
9308:S 05 Mar 2020 12:48:14.891 # Bio thread for job type #2 terminated
*** Preparing to test memory region 55c80e682000 (102400 bytes)
*** Preparing to test memory region 55c80f9a4000 (3665920 bytes)
*** Preparing to test memory region 55c80fd23000 (48634228736 bytes)
*** Preparing to test memory region 7f30abfff000 (1073745920 bytes)
*** Preparing to test memory region 7f30ec000000 (123854848 bytes)
*** Preparing to test memory region 7f30f4000000 (134217728 bytes)
*** Preparing to test memory region 7f30fc000000 (134217728 bytes)
*** Preparing to test memory region 7f3104000000 (134217728 bytes)
*** Preparing to test memory region 7f310c000000 (134217728 bytes)
*** Preparing to test memory region 7f3114000000 (134217728 bytes)
*** Preparing to test memory region 7f311c000000 (134217728 bytes)
*** Preparing to test memory region 7f3124000000 (134217728 bytes)
*** Preparing to test memory region 7f312c000000 (134217728 bytes)
*** Preparing to test memory region 7f3134000000 (134217728 bytes)
*** Preparing to test memory region 7f313c000000 (134217728 bytes)
*** Preparing to test memory region 7f3144000000 (134217728 bytes)
*** Preparing to test memory region 7f314c000000 (134217728 bytes)
*** Preparing to test memory region 7f3154000000 (134217728 bytes)
*** Preparing to test memory region 7f315c000000 (134217728 bytes)
*** Preparing to test memory region 7f3164000000 (134217728 bytes)
*** Preparing to test memory region 7f316c000000 (134217728 bytes)
*** Preparing to test memory region 7f3174000000 (134217728 bytes)
*** Preparing to test memory region 7f317c000000 (134217728 bytes)
*** Preparing to test memory region 7f3184000000 (134217728 bytes)
*** Preparing to test memory region 7f318c000000 (134217728 bytes)
*** Preparing to test memory region 7f3194000000 (134217728 bytes)
*** Preparing to test memory region 7f319c000000 (134217728 bytes)
*** Preparing to test memory region 7f31a4000000 (134217728 bytes)
*** Preparing to test memory region 7f31ac000000 (134217728 bytes)
*** Preparing to test memory region 7f31b4000000 (134217728 bytes)
*** Preparing to test memory region 7f31bc000000 (134217728 bytes)
*** Preparing to test memory region 7f31c4000000 (134217728 bytes)
*** Preparing to test memory region 7f31cc000000 (134217728 bytes)
*** Preparing to test memory region 7f31d4000000 (134217728 bytes)
*** Preparing to test memory region 7f31dc000000 (134217728 bytes)
*** Preparing to test memory region 7f31e4000000 (134217728 bytes)
*** Preparing to test memory region 7f31ec000000 (134217728 bytes)
*** Preparing to test memory region 7f31f4000000 (134217728 bytes)
*** Preparing to test memory region 7f31fc000000 (134217728 bytes)
*** Preparing to test memory region 7f3204000000 (134217728 bytes)
*** Preparing to test memory region 7f320c000000 (134217728 bytes)
*** Preparing to test memory region 7f3214000000 (134217728 bytes)
*** Preparing to test memory region 7f321c000000 (134217728 bytes)
*** Preparing to test memory region 7f3224000000 (134217728 bytes)
*** Preparing to test memory region 7f322c000000 (134217728 bytes)
*** Preparing to test memory region 7f3234000000 (134217728 bytes)
*** Preparing to test memory region 7f323c000000 (134217728 bytes)
*** Preparing to test memory region 7f3244000000 (134217728 bytes)
*** Preparing to test memory region 7f324c000000 (134217728 bytes)
*** Preparing to test memory region 7f3254000000 (134217728 bytes)
*** Preparing to test memory region 7f325c000000 (134217728 bytes)
*** Preparing to test memory region 7f3264000000 (134217728 bytes)
*** Preparing to test memory region 7f326c000000 (134217728 bytes)
*** Preparing to test memory region 7f3274000000 (134217728 bytes)
*** Preparing to test memory region 7f327c000000 (134217728 bytes)
*** Preparing to test memory region 7f3284000000 (134217728 bytes)
*** Preparing to test memory region 7f328c000000 (134217728 bytes)
*** Preparing to test memory region 7f3294000000 (134217728 bytes)
*** Preparing to test memory region 7f329c000000 (134217728 bytes)
*** Preparing to test memory region 7f32a4000000 (134217728 bytes)
*** Preparing to test memory region 7f32ac000000 (134217728 bytes)
*** Preparing to test memory region 7f32b4000000 (134217728 bytes)
*** Preparing to test memory region 7f32bc000000 (134217728 bytes)
*** Preparing to test memory region 7f32c4000000 (134217728 bytes)
*** Preparing to test memory region 7f32cc000000 (134217728 bytes)
*** Preparing to test memory region 7f32d4000000 (134217728 bytes)
*** Preparing to test memory region 7f32dc000000 (134217728 bytes)
*** Preparing to test memory region 7f32e4000000 (134217728 bytes)
*** Preparing to test memory region 7f32ec000000 (134217728 bytes)
*** Preparing to test memory region 7f32f4000000 (134217728 bytes)
*** Preparing to test memory region 7f32fc000000 (134217728 bytes)
*** Preparing to test memory region 7f3304000000 (134217728 bytes)
*** Preparing to test memory region 7f330c000000 (134217728 bytes)
*** Preparing to test memory region 7f3314000000 (134217728 bytes)
*** Preparing to test memory region 7f331c000000 (134217728 bytes)
*** Preparing to test memory region 7f3324000000 (134217728 bytes)
*** Preparing to test memory region 7f332c000000 (134217728 bytes)
*** Preparing to test memory region 7f3334000000 (134217728 bytes)
*** Preparing to test memory region 7f333c000000 (134217728 bytes)
*** Preparing to test memory region 7f3344000000 (134217728 bytes)
*** Preparing to test memory region 7f334c000000 (134217728 bytes)
*** Preparing to test memory region 7f3354000000 (134217728 bytes)
*** Preparing to test memory region 7f335c000000 (134217728 bytes)
*** Preparing to test memory region 7f3364000000 (134217728 bytes)
*** Preparing to test memory region 7f336c000000 (134217728 bytes)
*** Preparing to test memory region 7f3374000000 (134217728 bytes)
*** Preparing to test memory region 7f337c000000 (134217728 bytes)
*** Preparing to test memory region 7f3384000000 (134217728 bytes)
*** Preparing to test memory region 7f338c000000 (134217728 bytes)
*** Preparing to test memory region 7f3394000000 (134217728 bytes)
*** Preparing to test memory region 7f339c000000 (134217728 bytes)
*** Preparing to test memory region 7f33a4000000 (134217728 bytes)
*** Preparing to test memory region 7f33ac000000 (134217728 bytes)
*** Preparing to test memory region 7f33b4000000 (134217728 bytes)
*** Preparing to test memory region 7f33bc000000 (134217728 bytes)
*** Preparing to test memory region 7f33c4000000 (134217728 bytes)
*** Preparing to test memory region 7f33cc000000 (134217728 bytes)
*** Preparing to test memory region 7f33d4000000 (134217728 bytes)
*** Preparing to test memory region 7f33dc000000 (134217728 bytes)
*** Preparing to test memory region 7f33e4000000 (134217728 bytes)
*** Preparing to test memory region 7f33ec000000 (134217728 bytes)
*** Preparing to test memory region 7f33f4000000 (134217728 bytes)
*** Preparing to test memory region 7f33fc000000 (134217728 bytes)
*** Preparing to test memory region 7f3404000000 (134217728 bytes)
*** Preparing to test memory region 7f340c000000 (134217728 bytes)
*** Preparing to test memory region 7f3414000000 (134217728 bytes)
*** Preparing to test memory region 7f341c000000 (134217728 bytes)
*** Preparing to test memory region 7f3424000000 (134217728 bytes)
*** Preparing to test memory region 7f342c000000 (134217728 bytes)
*** Preparing to test memory region 7f3434000000 (134217728 bytes)
*** Preparing to test memory region 7f343c000000 (134217728 bytes)
*** Preparing to test memory region 7f3444000000 (134217728 bytes)
*** Preparing to test memory region 7f344c000000 (134217728 bytes)
*** Preparing to test memory region 7f3454000000 (134217728 bytes)
*** Preparing to test memory region 7f345c000000 (134217728 bytes)
*** Preparing to test memory region 7f3464000000 (134217728 bytes)
*** Preparing to test memory region 7f346c000000 (134217728 bytes)
*** Preparing to test memory region 7f3474000000 (134217728 bytes)
*** Preparing to test memory region 7f347c000000 (134217728 bytes)
*** Preparing to test memory region 7f3484000000 (134217728 bytes)
*** Preparing to test memory region 7f348c000000 (134217728 bytes)
*** Preparing to test memory region 7f3494000000 (134217728 bytes)
*** Preparing to test memory region 7f349c000000 (134217728 bytes)
*** Preparing to test memory region 7f34a4000000 (134217728 bytes)
*** Preparing to test memory region 7f34ac000000 (134217728 bytes)
*** Preparing to test memory region 7f34b4000000 (134217728 bytes)
*** Preparing to test memory region 7f34bc000000 (134217728 bytes)
*** Preparing to test memory region 7f34c4000000 (134217728 bytes)
*** Preparing to test memory region 7f34cc000000 (134217728 bytes)
*** Preparing to test memory region 7f34d4000000 (134217728 bytes)
*** Preparing to test memory region 7f34dc000000 (134217728 bytes)
*** Preparing to test memory region 7f34e4000000 (134217728 bytes)
*** Preparing to test memory region 7f34ec000000 (134217728 bytes)
*** Preparing to test memory region 7f34f4000000 (134217728 bytes)
*** Preparing to test memory region 7f34fc000000 (134217728 bytes)
*** Preparing to test memory region 7f3504000000 (134217728 bytes)
*** Preparing to test memory region 7f350c000000 (134217728 bytes)
*** Preparing to test memory region 7f3514000000 (134217728 bytes)
*** Preparing to test memory region 7f351c000000 (134217728 bytes)
*** Preparing to test memory region 7f3524000000 (134217728 bytes)
*** Preparing to test memory region 7f352c000000 (134217728 bytes)
*** Preparing to test memory region 7f3534000000 (134217728 bytes)
*** Preparing to test memory region 7f353c000000 (134217728 bytes)
*** Preparing to test memory region 7f3544000000 (134217728 bytes)
*** Preparing to test memory region 7f354c000000 (134217728 bytes)
*** Preparing to test memory region 7f3554000000 (134217728 bytes)
*** Preparing to test memory region 7f355c000000 (134217728 bytes)
*** Preparing to test memory region 7f3564000000 (134217728 bytes)
*** Preparing to test memory region 7f356c000000 (134217728 bytes)
*** Preparing to test memory region 7f3574000000 (134217728 bytes)
*** Preparing to test memory region 7f357c000000 (134217728 bytes)
*** Preparing to test memory region 7f3584000000 (134217728 bytes)
*** Preparing to test memory region 7f358c000000 (134217728 bytes)
*** Preparing to test memory region 7f3594000000 (134217728 bytes)
*** Preparing to test memory region 7f359c000000 (134217728 bytes)
*** Preparing to test memory region 7f35a4000000 (134217728 bytes)
*** Preparing to test memory region 7f35ac000000 (134217728 bytes)
*** Preparing to test memory region 7f35b4000000 (134217728 bytes)
*** Preparing to test memory region 7f35bc000000 (134217728 bytes)
*** Preparing to test memory region 7f35c4000000 (134217728 bytes)
*** Preparing to test memory region 7f35cc000000 (134217728 bytes)
*** Preparing to test memory region 7f35d4000000 (134217728 bytes)
*** Preparing to test memory region 7f35dc000000 (134217728 bytes)
*** Preparing to test memory region 7f35e4000000 (134217728 bytes)
*** Preparing to test memory region 7f35ec000000 (134217728 bytes)
*** Preparing to test memory region 7f35f4000000 (134217728 bytes)
*** Preparing to test memory region 7f35fc000000 (134217728 bytes)
*** Preparing to test memory region 7f3604000000 (134217728 bytes)
*** Preparing to test memory region 7f360c000000 (134217728 bytes)
*** Preparing to test memory region 7f3614000000 (134217728 bytes)
*** Preparing to test memory region 7f361c000000 (134217728 bytes)
*** Preparing to test memory region 7f3624000000 (134217728 bytes)
*** Preparing to test memory region 7f362c000000 (134217728 bytes)
*** Preparing to test memory region 7f3634000000 (134217728 bytes)
*** Preparing to test memory region 7f363c000000 (134217728 bytes)
*** Preparing to test memory region 7f3644000000 (134217728 bytes)
*** Preparing to test memory region 7f364c000000 (134217728 bytes)
*** Preparing to test memory region 7f3654000000 (134217728 bytes)
*** Preparing to test memory region 7f365c000000 (67108864 bytes)
*** Preparing to test memory region 7f3663fff000 (134221824 bytes)
*** Preparing to test memory region 7f366c000000 (134217728 bytes)
*** Preparing to test memory region 7f3674000000 (134217728 bytes)
*** Preparing to test memory region 7f367c000000 (134217728 bytes)
*** Preparing to test memory region 7f3684000000 (134217728 bytes)
*** Preparing to test memory region 7f368c000000 (134217728 bytes)
*** Preparing to test memory region 7f3694000000 (134217728 bytes)
*** Preparing to test memory region 7f369c000000 (134217728 bytes)
*** Preparing to test memory region 7f36a4000000 (134217728 bytes)
*** Preparing to test memory region 7f36ac000000 (134217728 bytes)
*** Preparing to test memory region 7f36b4000000 (134217728 bytes)
*** Preparing to test memory region 7f36bc000000 (134217728 bytes)
*** Preparing to test memory region 7f36c4000000 (134217728 bytes)
*** Preparing to test memory region 7f36cc000000 (134217728 bytes)
*** Preparing to test memory region 7f36d4000000 (134217728 bytes)
*** Preparing to test memory region 7f36dc000000 (134217728 bytes)
*** Preparing to test memory region 7f36e4000000 (134217728 bytes)
*** Preparing to test memory region 7f36ec000000 (134217728 bytes)
*** Preparing to test memory region 7f36f4000000 (134217728 bytes)
*** Preparing to test memory region 7f36fc000000 (134217728 bytes)
*** Preparing to test memory region 7f3704000000 (134217728 bytes)
*** Preparing to test memory region 7f370c000000 (134217728 bytes)
*** Preparing to test memory region 7f3714000000 (134217728 bytes)
*** Preparing to test memory region 7f371c000000 (134217728 bytes)
*** Preparing to test memory region 7f3724000000 (134217728 bytes)
*** Preparing to test memory region 7f372c000000 (134217728 bytes)
*** Preparing to test memory region 7f3734000000 (134217728 bytes)
*** Preparing to test memory region 7f373c000000 (134217728 bytes)
*** Preparing to test memory region 7f3744000000 (134217728 bytes)
*** Preparing to test memory region 7f374c000000 (134217728 bytes)
*** Preparing to test memory region 7f3754000000 (134217728 bytes)
*** Preparing to test memory region 7f375c000000 (134217728 bytes)
*** Preparing to test memory region 7f3764000000 (134217728 bytes)
*** Preparing to test memory region 7f376c000000 (134217728 bytes)
*** Preparing to test memory region 7f3774000000 (134217728 bytes)
*** Preparing to test memory region 7f377c000000 (134217728 bytes)
*** Preparing to test memory region 7f3784000000 (134217728 bytes)
*** Preparing to test memory region 7f378c000000 (134217728 bytes)
*** Preparing to test memory region 7f3794000000 (134217728 bytes)
*** Preparing to test memory region 7f379c000000 (134217728 bytes)
*** Preparing to test memory region 7f37a4000000 (134217728 bytes)
*** Preparing to test memory region 7f37ac000000 (134217728 bytes)
*** Preparing to test memory region 7f37b4000000 (134217728 bytes)
*** Preparing to test memory region 7f37bc000000 (134217728 bytes)
*** Preparing to test memory region 7f37c4000000 (134217728 bytes)
*** Preparing to test memory region 7f37cc000000 (134217728 bytes)
*** Preparing to test memory region 7f37d4000000 (134217728 bytes)
*** Preparing to test memory region 7f37dc000000 (134217728 bytes)
*** Preparing to test memory region 7f37e4000000 (134217728 bytes)
*** Preparing to test memory region 7f37ec000000 (134217728 bytes)
*** Preparing to test memory region 7f37f4000000 (134217728 bytes)
*** Preparing to test memory region 7f37fc000000 (134217728 bytes)
*** Preparing to test memory region 7f3804000000 (134217728 bytes)
*** Preparing to test memory region 7f380c000000 (134217728 bytes)
*** Preparing to test memory region 7f3814000000 (134217728 bytes)
*** Preparing to test memory region 7f381c000000 (134217728 bytes)
*** Preparing to test memory region 7f3824000000 (134217728 bytes)
*** Preparing to test memory region 7f382c000000 (134217728 bytes)
*** Preparing to test memory region 7f3834000000 (134217728 bytes)
*** Preparing to test memory region 7f383c000000 (134217728 bytes)
*** Preparing to test memory region 7f3844000000 (134217728 bytes)
*** Preparing to test memory region 7f384c000000 (134217728 bytes)
*** Preparing to test memory region 7f3854000000 (134217728 bytes)
*** Preparing to test memory region 7f385c000000 (134217728 bytes)
*** Preparing to test memory region 7f3864000000 (134217728 bytes)
*** Preparing to test memory region 7f386c000000 (134217728 bytes)
*** Preparing to test memory region 7f3874000000 (134217728 bytes)
*** Preparing to test memory region 7f387c000000 (134217728 bytes)
*** Preparing to test memory region 7f3884000000 (134217728 bytes)
*** Preparing to test memory region 7f388c000000 (134217728 bytes)
*** Preparing to test memory region 7f3894000000 (134217728 bytes)
*** Preparing to test memory region 7f389c000000 (134217728 bytes)
*** Preparing to test memory region 7f38a4000000 (134217728 bytes)
*** Preparing to test memory region 7f38ac000000 (134217728 bytes)
*** Preparing to test memory region 7f38b4000000 (134217728 bytes)
*** Preparing to test memory region 7f38bc000000 (134217728 bytes)
*** Preparing to test memory region 7f38c4000000 (134217728 bytes)
*** Preparing to test memory region 7f38cc000000 (134217728 bytes)
*** Preparing to test memory region 7f38d4000000 (134217728 bytes)
*** Preparing to test memory region 7f38dc000000 (134217728 bytes)
*** Preparing to test memory region 7f38e4000000 (134217728 bytes)
*** Preparing to test memory region 7f38ec000000 (134217728 bytes)
*** Preparing to test memory region 7f38f4000000 (134217728 bytes)
*** Preparing to test memory region 7f38fc000000 (134217728 bytes)
*** Preparing to test memory region 7f3904000000 (134217728 bytes)
*** Preparing to test memory region 7f390c000000 (134217728 bytes)
*** Preparing to test memory region 7f3914000000 (134217728 bytes)
*** Preparing to test memory region 7f391c000000 (134217728 bytes)
*** Preparing to test memory region 7f3924000000 (134217728 bytes)
*** Preparing to test memory region 7f392c000000 (134217728 bytes)
*** Preparing to test memory region 7f3934000000 (134217728 bytes)
*** Preparing to test memory region 7f393c000000 (134217728 bytes)
*** Preparing to test memory region 7f3944000000 (134217728 bytes)
*** Preparing to test memory region 7f394c000000 (134217728 bytes)
*** Preparing to test memory region 7f3954000000 (201326592 bytes)
*** Preparing to test memory region 7f3960000000 (134217728 bytes)
*** Preparing to test memory region 7f3968000000 (134217728 bytes)
*** Preparing to test memory region 7f3970000000 (134217728 bytes)
*** Preparing to test memory region 7f3978000000 (134217728 bytes)
*** Preparing to test memory region 7f3980000000 (134217728 bytes)
*** Preparing to test memory region 7f3988000000 (134217728 bytes)
*** Preparing to test memory region 7f3990000000 (134217728 bytes)
*** Preparing to test memory region 7f3998000000 (134217728 bytes)
*** Preparing to test memory region 7f39a0000000 (134217728 bytes)
*** Preparing to test memory region 7f39a8000000 (134217728 bytes)
*** Preparing to test memory region 7f39b0000000 (134217728 bytes)
*** Preparing to test memory region 7f39b8000000 (134217728 bytes)
*** Preparing to test memory region 7f39c0000000 (134217728 bytes)
*** Preparing to test memory region 7f39c8000000 (134217728 bytes)
*** Preparing to test memory region 7f39d0000000 (134217728 bytes)
*** Preparing to test memory region 7f39d8000000 (134217728 bytes)
*** Preparing to test memory region 7f39e0000000 (134217728 bytes)
*** Preparing to test memory region 7f39e8000000 (134217728 bytes)
*** Preparing to test memory region 7f39f0000000 (134217728 bytes)
*** Preparing to test memory region 7f39f8000000 (134217728 bytes)
*** Preparing to test memory region 7f3a00000000 (134217728 bytes)
*** Preparing to test memory region 7f3a08000000 (134217728 bytes)
*** Preparing to test memory region 7f3a10000000 (134217728 bytes)
*** Preparing to test memory region 7f3a18000000 (134217728 bytes)
*** Preparing to test memory region 7f3a20000000 (134217728 bytes)
*** Preparing to test memory region 7f3a28000000 (134217728 bytes)
*** Preparing to test memory region 7f3a30000000 (134217728 bytes)
*** Preparing to test memory region 7f3a38000000 (134217728 bytes)
*** Preparing to test memory region 7f3a40000000 (134217728 bytes)
*** Preparing to test memory region 7f3a48000000 (134217728 bytes)
*** Preparing to test memory region 7f3a50000000 (134217728 bytes)
*** Preparing to test memory region 7f3a58000000 (134217728 bytes)
*** Preparing to test memory region 7f3a60000000 (134217728 bytes)
*** Preparing to test memory region 7f3a68000000 (134217728 bytes)
*** Preparing to test memory region 7f3a70000000 (134217728 bytes)
*** Preparing to test memory region 7f3a78000000 (134217728 bytes)
*** Preparing to test memory region 7f3a80000000 (134217728 bytes)
*** Preparing to test memory region 7f3a88000000 (134217728 bytes)
*** Preparing to test memory region 7f3a90000000 (134217728 bytes)
*** Preparing to test memory region 7f3a98000000 (134217728 bytes)
*** Preparing to test memory region 7f3aa0000000 (134217728 bytes)
*** Preparing to test memory region 7f3aa8000000 (134217728 bytes)
*** Preparing to test memory region 7f3ab0000000 (134217728 bytes)
*** Preparing to test memory region 7f3ab8000000 (134217728 bytes)
*** Preparing to test memory region 7f3ac0000000 (134217728 bytes)
*** Preparing to test memory region 7f3ac8000000 (134217728 bytes)
*** Preparing to test memory region 7f3ad0000000 (134217728 bytes)
*** Preparing to test memory region 7f3ad8000000 (134217728 bytes)
*** Preparing to test memory region 7f3ae0000000 (134217728 bytes)
*** Preparing to test memory region 7f3ae8000000 (134217728 bytes)
*** Preparing to test memory region 7f3af0000000 (134217728 bytes)
*** Preparing to test memory region 7f3af8000000 (134217728 bytes)
*** Preparing to test memory region 7f3b00000000 (134217728 bytes)
*** Preparing to test memory region 7f3b08000000 (134217728 bytes)
*** Preparing to test memory region 7f3b10000000 (134217728 bytes)
*** Preparing to test memory region 7f3b18000000 (134217728 bytes)
*** Preparing to test memory region 7f3b20000000 (134217728 bytes)
*** Preparing to test memory region 7f3b28000000 (134217728 bytes)
*** Preparing to test memory region 7f3b30000000 (134217728 bytes)
*** Preparing to test memory region 7f3b38000000 (134217728 bytes)
*** Preparing to test memory region 7f3b40000000 (134217728 bytes)
*** Preparing to test memory region 7f3b48000000 (134217728 bytes)
*** Preparing to test memory region 7f3b50000000 (134217728 bytes)
*** Preparing to test memory region 7f3b58000000 (134217728 bytes)
*** Preparing to test memory region 7f3b60000000 (134217728 bytes)
*** Preparing to test memory region 7f3b68000000 (134217728 bytes)
*** Preparing to test memory region 7f3b70000000 (134217728 bytes)
*** Preparing to test memory region 7f3b78000000 (134217728 bytes)
*** Preparing to test memory region 7f3b80000000 (134217728 bytes)
*** Preparing to test memory region 7f3b88000000 (134217728 bytes)
*** Preparing to test memory region 7f3b90000000 (61014016 bytes)
*** Preparing to test memory region 7f3b94000000 (67108864 bytes)
*** Preparing to test memory region 7f3b98000000 (134217728 bytes)
*** Preparing to test memory region 7f3ba0000000 (134217728 bytes)
*** Preparing to test memory region 7f3ba8000000 (134217728 bytes)
*** Preparing to test memory region 7f3bb0000000 (134217728 bytes)
*** Preparing to test memory region 7f3bb8000000 (134217728 bytes)
*** Preparing to test memory region 7f3bc0000000 (134217728 bytes)
*** Preparing to test memory region 7f3bc8000000 (134217728 bytes)
*** Preparing to test memory region 7f3bd0000000 (134217728 bytes)
*** Preparing to test memory region 7f3bd8000000 (134217728 bytes)
*** Preparing to test memory region 7f3be0000000 (134217728 bytes)
*** Preparing to test memory region 7f3be8000000 (134217728 bytes)
*** Preparing to test memory region 7f3bf0000000 (134217728 bytes)
*** Preparing to test memory region 7f3bf8000000 (134217728 bytes)
*** Preparing to test memory region 7f3c00000000 (134217728 bytes)
*** Preparing to test memory region 7f3c08000000 (134217728 bytes)
*** Preparing to test memory region 7f3c10000000 (134217728 bytes)
*** Preparing to test memory region 7f3c18000000 (134217728 bytes)
*** Preparing to test memory region 7f3c20000000 (134217728 bytes)
*** Preparing to test memory region 7f3c28000000 (134217728 bytes)
*** Preparing to test memory region 7f3c30000000 (134217728 bytes)
*** Preparing to test memory region 7f3c38000000 (134217728 bytes)
*** Preparing to test memory region 7f3c40000000 (135168 bytes)
*** Preparing to test memory region 7f3c44000000 (135168 bytes)
*** Preparing to test memory region 7f3c48000000 (135168 bytes)
*** Preparing to test memory region 7f3c4c000000 (135168 bytes)
*** Preparing to test memory region 7f3c50000000 (135168 bytes)
*** Preparing to test memory region 7f3c54000000 (135168 bytes)
*** Preparing to test memory region 7f3c58000000 (135168 bytes)
*** Preparing to test memory region 7f3c5c000000 (135168 bytes)
*** Preparing to test memory region 7f3c60000000 (201326592 bytes)
*** Preparing to test memory region 7f3c6c000000 (67108864 bytes)
*** Preparing to test memory region 7f3c72e62000 (8388608 bytes)
*** Preparing to test memory region 7f3c73663000 (8388608 bytes)
*** Preparing to test memory region 7f3c73e64000 (8388608 bytes)
*** Preparing to test memory region 7f3c74665000 (8388608 bytes)
*** Preparing to test memory region 7f3c74e66000 (8388608 bytes)
*** Preparing to test memory region 7f3c75667000 (8388608 bytes)
*** Preparing to test memory region 7f3c75e68000 (8388608 bytes)
*** Preparing to test memory region 7f3c76669000 (4303360000 bytes)
*** Preparing to test memory region 7f3d76e6b000 (8388608 bytes)
*** Preparing to test memory region 7f3d7766c000 (8388608 bytes)
*** Preparing to test memory region 7f3d77e6d000 (8388608 bytes)
*** Preparing to test memory region 7f3d7866d000 (3276800 bytes)
*** Preparing to test memory region 7f3d79058000 (16384 bytes)
*** Preparing to test memory region 7f3d79277000 (16384 bytes)
*** Preparing to test memory region 7f3d79bb6000 (16384 bytes)
*** Preparing to test memory region 7f3d79ffb000 (1994752 bytes)
*** Preparing to test memory region 7f3d7a1ee000 (4096 bytes)
.9551:signal-handler (1583412496) Received SIGTERM scheduling shutdown...
The text was updated successfully, but these errors were encountered: