Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

redis server crashed with ft.search #728

Open
tomindo opened this issue Jun 11, 2019 · 5 comments

Comments

4 participants
@tomindo
Copy link

commented Jun 11, 2019

Please check the stack trace

=== REDIS BUG REPORT START: Cut & paste starting from here ===
20179:M 04 Jun 2019 18:58:34.646 # Redis 5.0.5 crashed by signal: 11
20179:M 04 Jun 2019 18:58:34.646 # Crashed running the instruction at: 0x7f8077280868
20179:M 04 Jun 2019 18:58:34.646 # Accessing address: 0x7f8077280868
20179:M 04 Jun 2019 18:58:34.646 # Failed assertion: <no assertion failed> (<no file>:0)

------ STACK TRACE ------
EIP:
/lib64/libc.so.6(+0x3c6868)[0x7f8077280868]

Backtrace:
/usr/bin/redis-server 0.0.0.0:6379(logStackTrace+0x29)[0x471049]
/usr/bin/redis-server 0.0.0.0:6379(sigsegvHandler+0xac)[0x4716ec]
/lib64/libpthread.so.0(+0xf5d0)[0x7f80772965d0]
/lib64/libc.so.6(+0x3c6868)[0x7f8077280868]

------ INFO OUTPUT ------
# Server
redis_version:5.0.5
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:225a3fe64a6e60b5
redis_mode:standalone
os:Linux 3.10.0-957.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:4.8.5
process_id:20179
run_id:e2c3e9c2d781e8b47ef67ab7c3e7851c9c5e9f91
tcp_port:6379
uptime_in_seconds:83188
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:16196042
executable:/usr/bin/redis-server
config_file:/var/run/redis/redis-6379.conf

# Clients
connected_clients:2
client_recent_max_input_buffer:2
client_recent_max_output_buffer:225544
blocked_clients:0

# Memory
used_memory:9014365160
used_memory_human:8.40G
used_memory_rss:9623519232
used_memory_rss_human:8.96G
used_memory_peak:9014579248
used_memory_peak_human:8.40G
used_memory_peak_perc:100.00%
used_memory_overhead:348377888
used_memory_startup:782544
used_memory_dataset:8665987272
used_memory_dataset_perc:96.14%
allocator_allocated:9016003960
allocator_active:9021210624
allocator_resident:9181880320
total_system_memory:101191393280
total_system_memory_human:94.24G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:25000000000
maxmemory_human:23.28G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.00
allocator_frag_bytes:5206664
allocator_rss_ratio:1.02
allocator_rss_bytes:160669696
rss_overhead_ratio:1.05
rss_overhead_bytes:441638912
mem_fragmentation_ratio:1.07
mem_fragmentation_bytes:609239120
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:87120
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1559692808
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:117
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:570720256
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

# Stats
total_connections_received:338
total_commands_processed:9361057
instantaneous_ops_per_sec:0
total_net_input_bytes:7599035636
total_net_output_bytes:471749830
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:3449
keyspace_misses:154
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:55302
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:master
connected_slaves:0
master_replid:2c2594a37ea39d12b8c39c3a5d1b5e125940950c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:553.272969
used_cpu_user:2092.916634
used_cpu_sys_children:30.925381
used_cpu_user_children:139.410338

# Commandstats
cmdstat_config:calls=5,usec=229,usec_per_call=45.80
cmdstat_info:calls=27,usec=2763,usec_per_call=102.33
cmdstat_select:calls=18,usec=102,usec_per_call=5.67
cmdstat_auth:calls=18,usec=29,usec_per_call=1.61
cmdstat_FT.GET:calls=4,usec=43,usec_per_call=10.75
cmdstat_monitor:calls=12,usec=10,usec_per_call=0.83
cmdstat_hgetall:calls=1883,usec=10671,usec_per_call=5.67
cmdstat_FT.SEARCH:calls=241,usec=23185646,usec_per_call=96206.00
cmdstat_ping:calls=960,usec=637,usec_per_call=0.66
cmdstat_FT.SUGGET:calls=4,usec=121,usec_per_call=30.25
cmdstat_keys:calls=1,usec=13,usec_per_call=13.00
cmdstat_FT.CREATE:calls=49,usec=13938,usec_per_call=284.45
cmdstat_FT.INFO:calls=13,usec=1050133,usec_per_call=80779.46
cmdstat_command:calls=148,usec=150777,usec_per_call=1018.76
cmdstat_memory:calls=4,usec=178,usec_per_call=44.50
cmdstat_module:calls=9,usec=2572,usec_per_call=285.78
cmdstat_FT.SUGADD:calls=4,usec=64,usec_per_call=16.00
cmdstat_del:calls=1949634,usec=2844038,usec_per_call=1.46
cmdstat_FT.DROP:calls=19,usec=21344566,usec_per_call=1123398.25
cmdstat_FT.ADD:calls=7408002,usec=574154129,usec_per_call=77.50
cmdstat_FT.EXPLAIN:calls=2,usec=52,usec_per_call=26.00

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=7009984,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------
id=2362753 addr=x.x.x.x:3238 fd=9 name= age=21613 idle=11486 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=FT.SEARCH
id=9390952 addr=127.0.0.1:30896 fd=8 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=129 qbuf-free=32639 obl=0 oll=1 omem=20504 events=r cmd=FT.SEARCH

------ CURRENT CLIENT INFO ------
id=9390952 addr=127.0.0.1:30896 fd=8 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=129 qbuf-free=32639 obl=0 oll=1 omem=20504 events=r cmd=FT.SEARCH
argv[0]: 'FT.SEARCH'
argv[1]: 'Ehann\RediSearch\Index'
argv[2]: 'pillow|bungalow'
argv[3]: 'INKEYS'
argv[4]: '3'
argv[5]: 'title'
argv[6]: 'description'
argv[7]: 'brand'

------ REGISTERS ------
20179:M 04 Jun 2019 18:58:34.647 # 
RAX:00007f8077280818 RBX:0000000000000000
RCX:0000000000000024 RDX:0000000000000018
RDI:00007f8077280818 RSI:000000000000000c
RBP:0000000000000000 RSP:00007fff46831d88
R8 :00007f8076a059e8 R9 :00007f8076c00900
R10:0000000000000000 R11:00007f8077046b80
R12:00007f80772807b8 R13:0000000000000008
R14:0000000000000001 R15:000000000000000f
RIP:00007f8077280868 EFL:0000000000010206
CSGSFS:0000000000000033
20179:M 04 Jun 2019 18:58:34.647 # (00007fff46831d97) -> 00007f8076a0aa00
20179:M 04 Jun 2019 18:58:34.647 # (00007fff46831d96) -> 00007f806e4e53d8
20179:M 04 Jun 2019 18:58:34.647 # (00007fff46831d95) -> 00007fff46831e50
20179:M 04 Jun 2019 18:58:34.647 # (00007fff46831d94) -> 00007fff46831e10
20179:M 04 Jun 2019 18:58:34.647 # (00007fff46831d93) -> 0000000000000000
20179:M 04 Jun 2019 18:58:34.647 # (00007fff46831d92) -> 00007f806e4e959c
20179:M 04 Jun 2019 18:58:34.647 # (00007fff46831d91) -> 00007f7daceef240
20179:M 04 Jun 2019 18:58:34.647 # (00007fff46831d90) -> 00007fff46831e50
20179:M 04 Jun 2019 18:58:34.647 # (00007fff46831d8f) -> 00000000012f0a70
20179:M 04 Jun 2019 18:58:34.647 # (00007fff46831d8e) -> 0000000000000000
20179:M 04 Jun 2019 18:58:34.647 # (00007fff46831d8d) -> 00000000012f0a70
20179:M 04 Jun 2019 18:58:34.647 # (00007fff46831d8c) -> 00007f806e520ec6
20179:M 04 Jun 2019 18:58:34.647 # (00007fff46831d8b) -> 00000000012f6760
20179:M 04 Jun 2019 18:58:34.647 # (00007fff46831d8a) -> 0000000000000001
20179:M 04 Jun 2019 18:58:34.647 # (00007fff46831d89) -> 00000000012f6760
20179:M 04 Jun 2019 18:58:34.648 # (00007fff46831d88) -> 00007f806e520e27

------ FAST MEMORY TEST ------
20179:M 04 Jun 2019 18:58:34.648 # Bio thread for job type #0 terminated
20179:M 04 Jun 2019 18:58:34.648 # Bio thread for job type #1 terminated
20179:M 04 Jun 2019 18:58:34.648 # Bio thread for job type #2 terminated
*** Preparing to test memory region 78e000 (2252800 bytes)
*** Preparing to test memory region 1294000 (520192 bytes)
*** Preparing to test memory region 7f7d98000000 (5521408 bytes)
*** Preparing to test memory region 7f7d9da00000 (3126853632 bytes)
*** Preparing to test memory region 7f7e58800000 (2273312768 bytes)
*** Preparing to test memory region 7f7ee0400000 (2076180480 bytes)
*** Preparing to test memory region 7f7f5c600000 (16777216 bytes)
*** Preparing to test memory region 7f7f5d800000 (8388608 bytes)
*** Preparing to test memory region 7f7f5e000000 (1845493760 bytes)
*** Preparing to test memory region 7f7fcc000000 (67244032 bytes)
*** Preparing to test memory region 7f7fd4000000 (135168 bytes)
*** Preparing to test memory region 7f7fd8000000 (135168 bytes)
*** Preparing to test memory region 7f7fdc000000 (135168 bytes)
*** Preparing to test memory region 7f7fe0000000 (135168 bytes)
*** Preparing to test memory region 7f7fe4000000 (135168 bytes)
*** Preparing to test memory region 7f7fe8000000 (135168 bytes)
*** Preparing to test memory region 7f7fec5fe000 (58720256 bytes)
*** Preparing to test memory region 7f7fefdff000 (8388608 bytes)
*** Preparing to test memory region 7f7ff0600000 (8388608 bytes)
*** Preparing to test memory region 7f7ff0e00000 (668991488 bytes)
*** Preparing to test memory region 7f8018d7d000 (607125504 bytes)
*** Preparing to test memory region 7f803d07e000 (8388608 bytes)
*** Preparing to test memory region 7f803d87f000 (8388608 bytes)
*** Preparing to test memory region 7f803e080000 (8388608 bytes)
*** Preparing to test memory region 7f803e880000 (22544384 bytes)
*** Preparing to test memory region 7f803ff80000 (101187584 bytes)
*** Preparing to test memory region 7f8046100000 (8388608 bytes)
*** Preparing to test memory region 7f8046900000 (91226112 bytes)
*** Preparing to test memory region 7f804c1fe000 (48234496 bytes)
*** Preparing to test memory region 7f804efff000 (8388608 bytes)
*** Preparing to test memory region 7f804f800000 (8388608 bytes)
*** Preparing to test memory region 7f8050000000 (135168 bytes)
*** Preparing to test memory region 7f80541fd000 (8388608 bytes)
*** Preparing to test memory region 7f80549fe000 (8388608 bytes)
*** Preparing to test memory region 7f80551ff000 (8388608 bytes)
*** Preparing to test memory region 7f8055a00000 (8388608 bytes)
*** Preparing to test memory region 7f8056200000 (4194304 bytes)
*** Preparing to test memory region 7f80567ff000 (8388608 bytes)
*** Preparing to test memory region 7f8057000000 (8388608 bytes)
*** Preparing to test memory region 7f8057800000 (4194304 bytes)
*** Preparing to test memory region 7f8057c79000 (8388608 bytes)
*** Preparing to test memory region 7f805847a000 (8388608 bytes)
*** Preparing to test memory region 7f8058c7b000 (8388608 bytes)
*** Preparing to test memory region 7f805947c000 (8388608 bytes)
*** Preparing to test memory region 7f8059c7d000 (8388608 bytes)
*** Preparing to test memory region 7f805a47e000 (8388608 bytes)
*** Preparing to test memory region 7f805ac7f000 (8388608 bytes)
*** Preparing to test memory region 7f805b480000 (8388608 bytes)
*** Preparing to test memory region 7f805bc81000 (8388608 bytes)
*** Preparing to test memory region 7f805c482000 (8388608 bytes)
*** Preparing to test memory region 7f805cc83000 (8388608 bytes)
*** Preparing to test memory region 7f805d484000 (8388608 bytes)
*** Preparing to test memory region 7f805dc85000 (8388608 bytes)
*** Preparing to test memory region 7f805e486000 (8388608 bytes)
*** Preparing to test memory region 7f805ec87000 (8388608 bytes)
*** Preparing to test memory region 7f805f488000 (8388608 bytes)
*** Preparing to test memory region 7f805fc89000 (8388608 bytes)
*** Preparing to test memory region 7f806048a000 (8388608 bytes)
*** Preparing to test memory region 7f8060c8b000 (8388608 bytes)
*** Preparing to test memory region 7f806148c000 (8388608 bytes)
*** Preparing to test memory region 7f8061c8d000 (8388608 bytes)
*** Preparing to test memory region 7f806248e000 (8388608 bytes)
*** Preparing to test memory region 7f8062c8f000 (8388608 bytes)
*** Preparing to test memory region 7f8063490000 (8388608 bytes)
*** Preparing to test memory region 7f8063c91000 (8388608 bytes)
*** Preparing to test memory region 7f8064492000 (8388608 bytes)
*** Preparing to test memory region 7f8064c93000 (8388608 bytes)
*** Preparing to test memory region 7f8065494000 (8388608 bytes)
*** Preparing to test memory region 7f8065c95000 (8388608 bytes)
*** Preparing to test memory region 7f8066496000 (8388608 bytes)
*** Preparing to test memory region 7f8066c97000 (8388608 bytes)
*** Preparing to test memory region 7f8067498000 (8388608 bytes)
*** Preparing to test memory region 7f8067c99000 (8388608 bytes)
*** Preparing to test memory region 7f806849a000 (8388608 bytes)
*** Preparing to test memory region 7f8068c9b000 (8388608 bytes)
*** Preparing to test memory region 7f806949c000 (8388608 bytes)
*** Preparing to test memory region 7f8069c9d000 (8388608 bytes)
*** Preparing to test memory region 7f806a49e000 (8388608 bytes)
*** Preparing to test memory region 7f806ac9f000 (8388608 bytes)
*** Preparing to test memory region 7f806b4a0000 (8388608 bytes)
*** Preparing to test memory region 7f806bca1000 (8388608 bytes)
*** Preparing to test memory region 7f806c4a2000 (8388608 bytes)
*** Preparing to test memory region 7f806cca3000 (8388608 bytes)
*** Preparing to test memory region 7f806d4a4000 (8388608 bytes)
*** Preparing to test memory region 7f806dca5000 (8388608 bytes)
*** Preparing to test memory region 7f806e8d0000 (12288 bytes)
*** Preparing to test memory region 7f806e8d4000 (8388608 bytes)
*** Preparing to test memory region 7f806f0d5000 (8388608 bytes)
*** Preparing to test memory region 7f806f8d6000 (8388608 bytes)
*** Preparing to test memory region 7f8076600000 (8388608 bytes)
*** Preparing to test memory region 7f8077282000 (20480 bytes)
*** Preparing to test memory region 7f807749f000 (16384 bytes)
*** Preparing to test memory region 7f8077dc6000 (20480 bytes)
*** Preparing to test memory region 7f8077dd0000 (8192 bytes)
*** Preparing to test memory region 7f8077dd4000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: /lib64/libc.so.6 (base 0x7f8076eba000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
------

=== REDIS BUG REPORT END. Make sure to include from START to END. ===
@mnunberg

This comment has been minimized.

Copy link
Collaborator

commented Jun 11, 2019

Does this crash consistently? I was hoping to get something from the stack trace (thank you for posting), but unfortunately it doesn't contain any usable information. If this is easily reproducible with your dataset, it might be helpful to run it through gdb and see what it says.

@tomindo

This comment has been minimized.

Copy link
Author

commented Jun 11, 2019

yes, it crashed consistently with that command

@MeirShpilraien

This comment has been minimized.

Copy link
Collaborator

commented Jun 12, 2019

@tomindo which version

@mnunberg

This comment has been minimized.

Copy link
Collaborator

commented Jun 12, 2019

He's using master, I think.

@tomindo Do you know how to use gdb?

If you don't, something like:

gdb -ex run --args /path/to/redis-server args...

When it crashes, do (in gdb)

thread apply all bt

and post the output. This might give us a better stack trace

@mnunberg

This comment has been minimized.

Copy link
Collaborator

commented Jun 27, 2019

@tomindo Are you still experiencing issues with this?

@gkorland gkorland added the bug label Jul 1, 2019

@gkorland gkorland added this to To do in 1.6 Jul 1, 2019

@K-Jo K-Jo removed this from To do in 1.6 Jul 15, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.