Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CRASH] /opt/redis-stack/lib/redisgraph.so(Query_EvalNode+0x22)[0x7fe92f1731c2] #2591

Closed
postpersonality opened this issue Sep 13, 2022 · 2 comments · Fixed by #2592
Closed
Labels

Comments

@postpersonality
Copy link

=== REDIS BUG REPORT START: Cut & paste starting from here ===
8:M 13 Sep 2022 14:30:17.039 # Redis 6.2.7 crashed by signal: 11, si_code: 1
8:M 13 Sep 2022 14:30:17.039 # Accessing address: 0x20
8:M 13 Sep 2022 14:30:17.039 # Crashed running the instruction at: 0x7fe92f1731c2

------ STACK TRACE ------
EIP:
/opt/redis-stack/lib/redisgraph.so(Query_EvalNode+0x22)[0x7fe92f1731c2]

Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7fe931500420]
/opt/redis-stack/lib/redisgraph.so(Query_EvalNode+0x22)[0x7fe92f1731c2]
/opt/redis-stack/lib/redisgraph.so(QAST_Iterate+0x52)[0x7fe92f173bf2]
/opt/redis-stack/lib/redisgraph.so(+0x128b79d)[0x7fe92f17a79d]
/opt/redis-stack/lib/redisgraph.so(RediSearch_GetResultsIterator+0x45)[0x7fe92f17a985]
/opt/redis-stack/lib/redisgraph.so(+0x203d79)[0x7fe92e0f2d79]
/opt/redis-stack/lib/redisgraph.so(+0x1fedc8)[0x7fe92e0eddc8]
/opt/redis-stack/lib/redisgraph.so(+0x202d5c)[0x7fe92e0f1d5c]
/opt/redis-stack/lib/redisgraph.so(+0x202a34)[0x7fe92e0f1a34]
/opt/redis-stack/lib/redisgraph.so(ExecutionPlan_Execute+0x50)[0x7fe92e0e66c0]
/opt/redis-stack/lib/redisgraph.so(+0x1f1423)[0x7fe92e0e0423]
/opt/redis-stack/lib/redisgraph.so(+0x23a09f)[0x7fe92e12909f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7fe9314f4609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7fe931419133]

------ REGISTERS ------
8:M 13 Sep 2022 14:30:17.041 # 
RAX:0000000000000000 RBX:00007fe92ceea350
RCX:0000000000000000 RDX:00007fe92ceea250
RDI:00007fe92ceea1e0 RSI:0000000000000000
RBP:00007fe92ceea280 RSP:00007fe92ceea140
R8 :00007fe92ceea240 R9 :00007fe91cc1e360
R10:00007fe92ceea250 R11:00007fe931000900
R12:00007fe9088321e0 R13:00007fe930e62180
R14:00007fe908832250 R15:00007fe92ceea240
RIP:00007fe92f1731c2 EFL:0000000000010246
CSGSFS:002b000000000033
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea14f) -> 00007fe9088321e0
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea14e) -> 00007fe92ceea280
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea14d) -> 00007fe92ceea350
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea14c) -> 00007fe930e62180
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea14b) -> f8c1f521322fb000
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea14a) -> 0000000000000010
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea149) -> 0000561b88728954
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea148) -> 00007fe92ceeaeb0
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea147) -> f8c1f521322fb000
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea146) -> 00007fe908a04800
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea145) -> f8c1f521322fb000
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea144) -> 00007fe908a00900
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea143) -> 0000000000000000
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea142) -> 00007fe92ceeaeb0
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea141) -> 0000000000000001
8:M 13 Sep 2022 14:30:17.041 # (00007fe92ceea140) -> 00007fe8fc0041a0

------ INFO OUTPUT ------
# Server
redis_version:6.2.7
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:6e59f9c01559b59f
redis_mode:standalone
os:Linux 5.15.0-47-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:9.4.0
process_id:8
process_supervised:no
run_id:d3adf8e9bd6f250f81579f4332c17d4eef2ccf66
tcp_port:6379
server_time_usec:1663079417038438
uptime_in_seconds:11
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:2135032
executable:/opt/redis-stack/bin/redis-server
config_file:
io_threads_active:0

# Clients
connected_clients:1
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:1
tracking_clients:0
clients_in_timeout_table:0

# Memory
used_memory:222632864
used_memory_human:212.32M
used_memory_rss:240033792
used_memory_rss_human:228.91M
used_memory_peak:247169200
used_memory_peak_human:235.72M
used_memory_peak_perc:90.07%
used_memory_overhead:1007312
used_memory_startup:1007176
used_memory_dataset:221625552
used_memory_dataset_perc:100.00%
allocator_allocated:223119584
allocator_active:224817152
allocator_resident:231534592
total_system_memory:14475534336
total_system_memory_human:13.48G
used_memory_lua:32768
used_memory_lua_human:32.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.01
allocator_frag_bytes:1697568
allocator_rss_ratio:1.03
allocator_rss_bytes:6717440
rss_overhead_ratio:1.04
rss_overhead_bytes:8499200
mem_fragmentation_ratio:1.08
mem_fragmentation_bytes:17893736
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1663079406
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:1
total_commands_processed:5
instantaneous_ops_per_sec:0
total_net_input_bytes:200
total_net_output_bytes:4613
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:16
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:3
total_writes_processed:2
io_threaded_reads_processed:0
io_threaded_writes_processed:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:57172f6d0854980015cf569f3c254cc93d5707dc
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.431299
used_cpu_user:4.289413
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.002802
used_cpu_user_main_thread:0.000000

# Modules
module:name=timeseries,ver=10616,api=1,filters=0,usedby=[],using=[],options=[handle-io-errors]
module:name=ReJSON,ver=20009,api=1,filters=0,usedby=[search|graph],using=[],options=[handle-io-errors]
module:name=search,ver=20409,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
module:name=graph,ver=20815,api=1,filters=0,usedby=[],using=[ReJSON],options=[]
module:name=bf,ver=20217,api=1,filters=0,usedby=[],using=[],options=[]

# Commandstats
cmdstat_client:calls=1,usec=3,usec_per_call=3.00,rejected_calls=0,failed_calls=0
cmdstat_info:calls=3,usec=79,usec_per_call=26.33,rejected_calls=0,failed_calls=0
cmdstat_graph.QUERY:calls=1,usec=29,usec_per_call=29.00,rejected_calls=0,failed_calls=0

# Errorstats

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=1,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------
id=16 addr=172.18.0.1:48934 laddr=172.18.0.2:6379 fd=12 name=redisinsight-workbench-a8c7d442 age=1 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=94 obl=0 oll=0 omem=0 tot-mem=61558 events=r cmd=graph.QUERY user=default redir=-1

------ MODULES INFO OUTPUT ------
# ReJSON_trace
ReJSON_trace:   0: redis_module::base_info_func
   1: rejson::__info_func
   2: modulesCollectInfo
             at /__w/redis-stack/redis-stack/redis/src/module.c:7100:9
   3: logModulesInfo
             at /__w/redis-stack/redis-stack/redis/src/debug.c:1598:22
   4: printCrashReport
             at /__w/redis-stack/redis-stack/redis/src/debug.c:1849:5
   5: sigsegvHandler
             at /__w/redis-stack/redis-stack/redis/src/debug.c:1831:5
   6: <unknown>
   7: Query_EvalNode
   8: QAST_Iterate
   9: handleIterCommon
  10: RediSearch_GetResultsIterator
  11: IndexScanConsume
  12: CondTraverseConsume
  13: ProjectConsume
  14: ResultsConsume
  15: ExecutionPlan_Execute
  16: _ExecuteQuery
  17: thread_do
  18: start_thread
  19: clone


# graph_executing commands
graph_command:GRAPH.QUERY MATCH (a:Airport)-[f:Flight]->(b:Airport) WHERE id(a) IN [1,2,3] RETURN b

------ FAST MEMORY TEST ------
8:M 13 Sep 2022 14:30:17.051 # main thread terminated
8:M 13 Sep 2022 14:30:17.051 # Bio thread for job type #0 terminated
8:M 13 Sep 2022 14:30:17.051 # Bio thread for job type #1 terminated
8:M 13 Sep 2022 14:30:17.051 # Bio thread for job type #2 terminated

Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: Query_EvalNode (base: 0x7fe92f1731a0)
Module: /opt/redis-stack/lib/redisgraph.so (base 0x7fe92deef000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x7fe92f1731a0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
8:M 13 Sep 2022 14:30:17.051 # dump of function (hexdump of 162 bytes):
f30f1efa415741564155415455534883ec6864488b042528000000488944245831c0837e200e0f87f40200008b4620488d15d6d310004989fc4889f5486304824801d03effe0662e0f1f840000000000e80b8307004989c50f1f840000000000488b4424586448330425280000000f85650800004883c4684c89e85b5d415c415d415e415fc3662e0f1f840000000000e89bfeffff4989c5ebc6660f1f440000488b
Function at 0x7fe92f1eb500 is NewEmptyIterator
Function at 0x7fe92f1730d0 is Query_EvalTokenNode

=== REDIS BUG REPORT END. Make sure to include from START to END. ===
@gkorland gkorland added the bug label Sep 13, 2022
@gkorland
Copy link
Contributor

@postpersonality thanks for opening the issue, do you have a way to recreate the issue?

@postpersonality
Copy link
Author

Thanks for the blazing fast fix! Do you still need details for recreation?

@LiorKogan LiorKogan mentioned this issue Nov 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants