Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash while enforcing unique constraint on string property of nodes. #664

Closed
linuxfreakus opened this issue Apr 29, 2024 · 4 comments · Fixed by #667 or #663
Closed

Crash while enforcing unique constraint on string property of nodes. #664

linuxfreakus opened this issue Apr 29, 2024 · 4 comments · Fixed by #667 or #663
Assignees
@swilly22
Labels
bug Something isn't working

Comments

@linuxfreakus
Copy link

Summary

Currently we are running FalkorDB 4.0.9 on Kubernetes in an HA sentinel mode setup and we have run into a problem with adding a unique constraint on node properties.

We have nodes that have a URL property on them. Some of the URLs are pretty wonky and invalid, but nevertheless the strings made their way into the DB, and we desire to have no duplicate strings. There is already an index on the property because they are searched, but when we tried to add a unique constraint, it instantly crashes the database as well as all the replicas in our HA sentinel setup.

It appears that the constraint is added... and then replicated to the other nodes even though it is not operational yet... it then tries to enforce that constraint and crashes during the validation process, and the same thing happens on all other nodes, and unless there is intervention, the nodes will all continue to crash over and over since the constraint is in the AOF incremental file and it becomes necessary to either restore the PVC from backup, or modify the incr file to remove the offending issue.

It is not necessary to be using HA nodes to see this crash, but it is worth noting that the crash does take down the entire StatefulSet with it, so that might want to be considered when making a fix since it would be desirable to limit blast damage of things that resemble this situation (perhaps a config flag or something to control whether to replicate indices or constraints when they are not yet operational).

Also note that depending on what exactly is happening here in the code, it seems possible this could be exploited for denial of service or other malicious activity via specially crafted values (speculative, we have not tried to exploit).

Bug Report

=== REDIS BUG REPORT START: Cut & paste starting from here ===
1:M 26 Apr 2024 23:57:25.602 # Redis 7.2.4 crashed by signal: 11, si_code: 1
1:M 26 Apr 2024 23:57:25.602 # Accessing address: (nil)
1:M 26 Apr 2024 23:57:25.602 # Crashed running the instruction at: 0x7ff392f05dda

------ STACK TRACE ------
EIP:
/FalkorDB/bin/src/falkordb.so(EnforceUniqueEntity+0x17a)[0x7ff392f05dda]

Backtrace:
/lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7ff395894050]
/FalkorDB/bin/src/falkordb.so(EnforceUniqueEntity+0x17a)[0x7ff392f05dda]
/FalkorDB/bin/src/falkordb.so(Constraint_EnforceNodes+0xb3)[0x7ff392f055c3]
/FalkorDB/bin/src/falkordb.so(+0x2d1852)[0x7ff392f3b852]
/lib/x86_64-linux-gnu/libc.so.6(+0x89134)[0x7ff3958e1134]
/lib/x86_64-linux-gnu/libc.so.6(__clone+0x40)[0x7ff395960a40]

------ REGISTERS ------
1:M 26 Apr 2024 23:57:25.603 # 
RAX:0000000000000000 RBX:000000000011a1d4
RCX:0000000000000004 RDX:0000000000000000
RDI:0000000000000000 RSI:00007febb9a76008
RBP:00007febb9a76000 RSP:00007ff330d41230
R8 :00007feecce6cc80 R9 :00007feeccf7ff2b
R10:0000000000000002 R11:00007ff394b286e0
R12:00007ff330d41290 R13:0000000000000000
R14:00007feeea9e6180 R15:00007ff39542cda0
RIP:00007ff392f05dda EFL:0000000000010202
CSGSFS:002b000000000033
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d4123f) -> 000000000011a1d3
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d4123e) -> 00007fef96421000
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d4123d) -> 000000000011a1d4
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d4123c) -> 00007feecd1e6085
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d4123b) -> 000000000011a1d4
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d4123a) -> 00007febb9a19c20
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d41239) -> 0000000000200835
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d41238) -> 00000000000000d8
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d41237) -> 00007ff392f055c3
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d41236) -> 00007ff330d41288
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d41235) -> 00007ff330d412a0
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d41234) -> 00007fefbf4a0cc0
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d41233) -> 00007ff330d41290
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d41232) -> 00007ff39542cda0
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d41231) -> 000000000000116e
1:M 26 Apr 2024 23:57:25.603 # (00007ff330d41230) -> ffffffffffffffff

------ INFO OUTPUT ------
# Server
redis_version:7.2.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:7c3cbaf27e881543
redis_mode:standalone
os:Linux 5.15.0-92-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:12.2.0
process_id:1
process_supervised:no
run_id:95446247103c86816b2a0fe3978a86ae1a93f105
tcp_port:6379
server_time_usec:1714175845521484
uptime_in_seconds:3805
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:2899813
executable:/FalkorDB/redis-server
config_file:
io_threads_active:0
listener0:name=tcp,bind=*,bind=-::*,port=6379

# Clients
connected_clients:8
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:20504
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0

# Memory
used_memory:17121270144
used_memory_human:15.95G
used_memory_rss:21607366656
used_memory_rss_human:20.12G
used_memory_peak:29626500536
used_memory_peak_human:27.59G
used_memory_peak_perc:57.79%
used_memory_overhead:1801700
used_memory_startup:943688
used_memory_dataset:17119468444
used_memory_dataset_perc:99.99%
allocator_allocated:17123971016
allocator_active:22166970368
allocator_resident:22615248896
total_system_memory:1622922797056
total_system_memory_human:1.48T
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.29
allocator_frag_bytes:5042999352
allocator_rss_ratio:1.02
allocator_rss_bytes:448278528
rss_overhead_ratio:0.96
rss_overhead_bytes:-1007882240
mem_fragmentation_ratio:1.26
mem_fragmentation_bytes:4486099072
mem_not_counted_for_evict:512
mem_replication_backlog:717644
mem_total_replication_buffers:717640
mem_clients_slaves:0
mem_clients_normal:139448
mem_cluster_links:0
mem_aof_buffer:512
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:294
rdb_bgsave_in_progress:0
rdb_last_save_time:1714172040
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:52
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:11296768
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:921
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:66
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:1
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:2552492032
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:4644585097
aof_base_size:4644584204
aof_pending_rewrite:0
aof_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0

# Stats
total_connections_received:1679
total_commands_processed:27850
instantaneous_ops_per_sec:8
total_net_input_bytes:4646136690
total_net_output_bytes:9316400068
total_net_repl_input_bytes:4644672220
total_net_repl_output_bytes:9290323413
instantaneous_input_kbps:0.36
instantaneous_output_kbps:4.51
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.34
rejected_connections:0
sync_full:2
sync_partial_ok:0
sync_partial_err:2
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:39
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:19
keyspace_misses:1
pubsub_channels:1
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:512327
total_forks:3
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:1703
dump_payload_sanitizations:0
total_reads_processed:27082
total_writes_processed:42517
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:966
reply_buffer_expands:1061
eventloop_cycles:1344919
eventloop_duration_sum:448501292
eventloop_duration_cmd_sum:618847
instantaneous_eventloop_cycles_per_sec:20
instantaneous_eventloop_duration_usec:415
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:master
connected_slaves:2
slave0:ip=10.1.94.90,port=6379,state=online,offset=1084412,lag=0
slave1:ip=10.1.194.133,port=6379,state=online,offset=1084274,lag=2
master_failover_state:no-failover
master_replid:3bb67f41bbe6361a41ede7c59c40e1910104c3e1
master_replid2:d89741a52b9a71364676179b4c70edb16e1653f5
master_repl_offset:1084552
second_repl_offset:461982
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:374139
repl_backlog_histlen:710414

# CPU
used_cpu_sys:412.075794
used_cpu_user:895.024742
used_cpu_sys_children:13.511328
used_cpu_user_children:156.177694
used_cpu_sys_main_thread:0.752084
used_cpu_user_main_thread:2.864320

# Modules
module:name=graph,ver=40009,api=1,filters=0,usedby=[],using=[],options=[]

# Commandstats
cmdstat_graph.CONSTRAINT:calls=5,usec=744,usec_per_call=148.80,rejected_calls=0,failed_calls=0
cmdstat_multi:calls=64,usec=52,usec_per_call=0.81,rejected_calls=0,failed_calls=0
cmdstat_cluster|info:calls=1,usec=4,usec_per_call=4.00,rejected_calls=0,failed_calls=1
cmdstat_slowlog|len:calls=127,usec=206,usec_per_call=1.62,rejected_calls=0,failed_calls=0
cmdstat_slowlog|get:calls=127,usec=168,usec_per_call=1.32,rejected_calls=0,failed_calls=0
cmdstat_graph.RO_QUERY:calls=9,usec=4507,usec_per_call=500.78,rejected_calls=0,failed_calls=1
cmdstat_command|docs:calls=1,usec=1808,usec_per_call=1808.00,rejected_calls=0,failed_calls=0
cmdstat_graph.QUERY:calls=4,usec=2634,usec_per_call=658.50,rejected_calls=0,failed_calls=2
cmdstat_graph.LIST:calls=3,usec=27,usec_per_call=9.00,rejected_calls=0,failed_calls=0
cmdstat_slaveof:calls=2,usec=383,usec_per_call=191.50,rejected_calls=62,failed_calls=0
cmdstat_psync:calls=2,usec=62,usec_per_call=31.00,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=127,usec=28799,usec_per_call=226.76,rejected_calls=0,failed_calls=0
cmdstat_config|rewrite:calls=2,usec=15,usec_per_call=7.50,rejected_calls=0,failed_calls=2
cmdstat_replconf:calls=4740,usec=8140,usec_per_call=1.72,rejected_calls=0,failed_calls=0
cmdstat_publish:calls=6829,usec=42676,usec_per_call=6.25,rejected_calls=0,failed_calls=0
cmdstat_subscribe:calls=11,usec=49,usec_per_call=4.45,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=1679,usec=7029,usec_per_call=4.19,rejected_calls=0,failed_calls=0
cmdstat_latency|latest:calls=127,usec=180,usec_per_call=1.42,rejected_calls=0,failed_calls=0
cmdstat_latency|histogram:calls=127,usec=55874,usec_per_call=439.95,rejected_calls=0,failed_calls=0
cmdstat_info:calls=3476,usec=454802,usec_per_call=130.84,rejected_calls=0,failed_calls=0
cmdstat_client|setinfo:calls=6,usec=5,usec_per_call=0.83,rejected_calls=0,failed_calls=0
cmdstat_client|getredir:calls=1,usec=4,usec_per_call=4.00,rejected_calls=0,failed_calls=0
cmdstat_client|info:calls=2,usec=51,usec_per_call=25.50,rejected_calls=0,failed_calls=0
cmdstat_client|getname:calls=1,usec=3,usec_per_call=3.00,rejected_calls=0,failed_calls=0
cmdstat_client|setname:calls=147,usec=280,usec_per_call=1.90,rejected_calls=0,failed_calls=0
cmdstat_client|id:calls=1,usec=3,usec_per_call=3.00,rejected_calls=0,failed_calls=0
cmdstat_client|kill:calls=4,usec=314,usec_per_call=78.50,rejected_calls=0,failed_calls=0
cmdstat_exec:calls=64,usec=819,usec_per_call=12.80,rejected_calls=0,failed_calls=62
cmdstat_ping:calls=10159,usec=16747,usec_per_call=1.65,rejected_calls=1572,failed_calls=0
cmdstat_select:calls=2,usec=2,usec_per_call=1.00,rejected_calls=0,failed_calls=0

# Errorstats
errorstat_Attribute:count=2
errorstat_ERR:count=4
errorstat_EXECABORT:count=62
errorstat_LOADING:count=1632
errorstat_NOAUTH:count=2
errorstat_errMsg_:count=1

# Latencystats
latency_percentiles_usec_graph.CONSTRAINT:p50=63.231,p99=501.759,p99.9=501.759
latency_percentiles_usec_multi:p50=1.003,p99=3.007,p99.9=6.015
latency_percentiles_usec_cluster|info:p50=4.015,p99=4.015,p99.9=4.015
latency_percentiles_usec_slowlog|len:p50=2.007,p99=3.007,p99.9=3.007
latency_percentiles_usec_slowlog|get:p50=1.003,p99=2.007,p99.9=2.007
latency_percentiles_usec_graph.RO_QUERY:p50=491.519,p99=598.015,p99.9=598.015
latency_percentiles_usec_command|docs:p50=1810.431,p99=1810.431,p99.9=1810.431
latency_percentiles_usec_graph.QUERY:p50=573.439,p99=888.831,p99.9=888.831
latency_percentiles_usec_graph.LIST:p50=9.023,p99=11.007,p99.9=11.007
latency_percentiles_usec_slaveof:p50=187.391,p99=196.607,p99.9=196.607
latency_percentiles_usec_psync:p50=31.103,p99=31.103,p99.9=31.103
latency_percentiles_usec_config|get:p50=257.023,p99=323.583,p99.9=387.071
latency_percentiles_usec_config|rewrite:p50=7.007,p99=8.031,p99.9=8.031
latency_percentiles_usec_replconf:p50=2.007,p99=3.007,p99.9=3.007
latency_percentiles_usec_publish:p50=8.031,p99=13.055,p99.9=27.007
latency_percentiles_usec_subscribe:p50=4.015,p99=11.007,p99.9=11.007
latency_percentiles_usec_auth:p50=4.015,p99=9.023,p99.9=16.063
latency_percentiles_usec_latency|latest:p50=1.003,p99=2.007,p99.9=5.023
latency_percentiles_usec_latency|histogram:p50=485.375,p99=774.143,p99.9=786.431
latency_percentiles_usec_info:p50=157.695,p99=516.095,p99.9=638.975
latency_percentiles_usec_client|setinfo:p50=1.003,p99=2.007,p99.9=2.007
latency_percentiles_usec_client|getredir:p50=4.015,p99=4.015,p99.9=4.015
latency_percentiles_usec_client|info:p50=24.063,p99=27.007,p99.9=27.007
latency_percentiles_usec_client|getname:p50=3.007,p99=3.007,p99.9=3.007
latency_percentiles_usec_client|setname:p50=2.007,p99=3.007,p99.9=3.007
latency_percentiles_usec_client|id:p50=3.007,p99=3.007,p99.9=3.007
latency_percentiles_usec_client|kill:p50=38.143,p99=166.911,p99.9=166.911
latency_percentiles_usec_exec:p50=1.003,p99=266.239,p99.9=514.047
latency_percentiles_usec_ping:p50=2.007,p99=3.007,p99.9=10.047
latency_percentiles_usec_select:p50=1.003,p99=1.003,p99.9=1.003

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=4,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------
id=679 addr=10.1.194.133:36186 laddr=10.1.85.247:6379 fd=15 name=sentinel-39614331-cmd age=3013 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=2048 rbp=1024 obl=0 oll=0 omem=0 tot-mem=23424 events=r cmd=publish user=default redir=-1 resp=2 lib-name= lib-ver=
id=996 addr=10.1.194.133:58950 laddr=10.1.85.247:6379 fd=16 name= age=2646 idle=0 flags=S db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=1 omem=20504 tot-mem=42904 events=r cmd=replconf user=default redir=-1 resp=2 lib-name= lib-ver=
id=2520 addr=10.1.165.10:37144 laddr=10.1.85.247:6379 fd=18 name= age=866 idle=741 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=client|info user=default redir=-1 resp=2 lib-name= lib-ver=
id=668 addr=10.1.94.90:49510 laddr=10.1.85.247:6379 fd=8 name= age=3019 idle=0 flags=S db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=1 omem=20504 tot-mem=42904 events=r cmd=replconf user=default redir=-1 resp=2 lib-name= lib-ver=
id=827 addr=10.1.94.90:54600 laddr=10.1.85.247:6379 fd=13 name=sentinel-11446d3e-cmd age=2839 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=7 obl=0 oll=0 omem=0 tot-mem=22400 events=r cmd=ping user=default redir=-1 resp=2 lib-name= lib-ver=
id=828 addr=10.1.94.90:54610 laddr=10.1.85.247:6379 fd=14 name=sentinel-11446d3e-pubsub age=2839 idle=0 flags=P db=0 sub=1 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=140 obl=0 oll=0 omem=0 tot-mem=22456 events=r cmd=subscribe user=default redir=-1 resp=2 lib-name= lib-ver=
id=669 addr=10.1.85.247:48636 laddr=10.1.85.247:6379 fd=9 name=sentinel-0249ba83-cmd age=3019 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=7 obl=0 oll=0 omem=0 tot-mem=22400 events=r cmd=ping user=default redir=-1 resp=2 lib-name= lib-ver=
id=670 addr=10.1.85.247:48646 laddr=10.1.85.247:6379 fd=10 name=sentinel-0249ba83-pubsub age=3019 idle=0 flags=P db=0 sub=1 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=140 obl=0 oll=0 omem=0 tot-mem=22456 events=r cmd=subscribe user=default redir=-1 resp=2 lib-name= lib-ver=
id=2824 addr=10.1.94.71:52240 laddr=10.1.85.247:6379 fd=19 name= age=516 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=graph.CONSTRAINT user=default redir=-1 resp=2 lib-name=FalkorDB lib-ver=1.0.0
id=678 addr=10.1.194.133:36198 laddr=10.1.85.247:6379 fd=11 name=sentinel-39614331-pubsub age=3013 idle=0 flags=P db=0 sub=1 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=140 obl=0 oll=0 omem=0 tot-mem=22456 events=r cmd=subscribe user=default redir=-1 resp=2 lib-name= lib-ver=

------ MODULES INFO OUTPUT ------
# graph_executing commands

------ CONFIG DEBUG OUTPUT ------
sanitize-dump-payload no
proto-max-bulk-len 512mb
io-threads-do-reads no
lazyfree-lazy-user-flush no
repl-diskless-load disabled
lazyfree-lazy-server-del no
lazyfree-lazy-user-del no
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
client-query-buffer-limit 1gb
list-compress-depth 0
slave-read-only yes
activedefrag no
io-threads 1
replica-read-only yes
repl-diskless-sync yes

------ FAST MEMORY TEST ------
1:M 26 Apr 2024 23:57:25.605 # main thread terminated
1:M 26 Apr 2024 23:57:25.605 # Bio worker thread #0 terminated
1:M 26 Apr 2024 23:57:25.605 # Bio worker thread #1 terminated
1:M 26 Apr 2024 23:57:25.605 # Bio worker thread #2 terminated

Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: EnforceUniqueEntity (base: 0x7ff392f05c60)
Module: /FalkorDB/bin/src/falkordb.so (base 0x7ff392c6a000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x7ff392f05c60 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
1:M 26 Apr 2024 23:57:25.606 # dump of function (hexdump of 506 bytes):
f30f1efa41574989ff4889f7415641554989d541544989f455534883ec08e82d78020041803f00745f488b2d686c08024889c34531f6eb1e0f1f840000000000f74008ff87ffff7527410fb60f4183c6014438f1763a498b5728410fb6c64889df0fb73442e8266802004839450075d041bc010000004883c4084489e05b5d415c415d415e415fc331c9660f1f4400004d8b7748498b57284c89e64c89f7e86d4f03004c89f74889c5e8422e03004889ef4889c64989c6e85427a4014c89ff4889c5e8c9f4ffff31d24c89f64889ef83f8010f8498000000e8231fa401498b5c2408483958100f859700000031d24c89f64889efe8071fa4014885c0410f94c40f95c34889efe8e51fa4014d85ed0f8462ffffff84db0f845affffffe8df0d0400418b772031d241837f40014889c70f95c2e8d9bd02004c89ff4889c5e84ef4ffff4889ef83f8017446e8414c0400488d15c211be014889c1be010000004c89ef31c04531e4e8457efcffe906ffffffe88b1ea401498b5c24084839180f8469ffffffbb010000004531e4e973ffffffe8fb4b0400488d154c11be014889c1ebb866662e0f1f8400000000000f1f4000f30f1efa4157488d0523feffff415666480f6ec84589c641554d89cd41545589cd534889d34883ec384c8b3d185c0802f30f7e05886008024889742428660f6cc1897c2414bf50000000
Function at 0x7ff392f2d4b0 is GraphEntity_GetAttributes
Function at 0x7ff392f2c4f0 is AttributeSet_Get
Function at 0x7ff392f3ac70 is Index_BuildUniqueConstraintQuery
Function at 0x7ff392f38b50 is Index_RSIndex
Function at 0x7ff394948470 is RediSearch_GetResultsIterator
Function at 0x7ff394947c60 is RediSearch_ResultsIteratorNext
Function at 0x7ff394947d50 is RediSearch_ResultsIteratorFree
Function at 0x7ff392f46b60 is QueryCtx_GetGraphCtx
Function at 0x7ff392f31b70 is GraphContext_GetSchemaByID
Function at 0x7ff392f4a9f0 is Schema_GetName
Function at 0x7ff394947c60 is RediSearch_ResultsIteratorNext

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

Steps to Reproduce

  • Start an empty FalkorDB
  • Add some labeled nodes (i.e. Page) with string property (i.e. url) containing data that will trigger this issue (see attached)
  • Add an index on the property: GRAPH.QUERY page "CREATE INDEX FOR (p:Page) ON (p.url)"
  • Add the constraint on the indexed property: GRAPH.CONSTRAINT CREATE page UNIQUE NODE Page PROPERTIES 1 url
  • Observe the crash

Attachments

dump.rdb.gz
@alan-salimov-vai
Copy link

Related to this issue, we start from a fresh falkordb instance (pulled from the latest image as of this morning).

Inserting a single node of type "Page" then adding additional nodes normally works.

However, if you add and drop a constraint on the URL field, undefined behavior occurs. Either a segfault or the inserted URLs are malformed (resulting in UnicodeDecodeErrors when attempting to access.)

from falkordb import FalkorDB, Node
from redis.exceptions import ResponseError

def reset_graph(use_constraints=False, port=8080):
    # Connect to FalkorDB
    db = FalkorDB(host='localhost', port=port)
    graph = db.select_graph('Page')
    try:
        graph.delete()
    except ResponseError:
        pass
    for index, url in enumerate(['https://en.wikipedia.org/wiki/Napoleon']):
        node = Node(labels="Page", properties={'url':url})
        query = 'CREATE ' + (str(node))
        graph.query(query)
        
        if use_constraints:
            graph.create_node_unique_constraint('Page', 'url')
            graph.list_constraints()
            graph.drop_node_unique_constraint('Page', 'url')
        
    ct = graph.query("MATCH (n:Page) RETURN COUNT(n)").result_set[0][0]
    print("Graph reset, count=", ct)
    return graph

target_url_list = ['https://en.wikipedia.org/wiki/Assembly_language', 'https://en.wikipedia.org/wiki/Sabbatical']

# ---- NO CONSTRAINT SET ---- # 
graph = reset_graph(use_constraints=False)
insert_query = graph.query("""
        WITH $target_urls AS target_urls_list
        UNWIND target_urls_list AS target_url
        MATCH (src:Page {url: $source_url})
        MERGE (tgt:Page {url: target_url})
        MERGE (src)-[:CONNECTED_TO]->(tgt)
        RETURN src
    """, params={"source_url": 'https://en.wikipedia.org/wiki/Napoleon', "target_urls": target_url_list})

ct = graph.query("MATCH (n:Page) RETURN COUNT(n)").result_set[0][0]
print(ct) # Prints 3
url_match_set = graph.query("MATCH (n:Page) RETURN n.url").result_set # Returns 3 URLs 

# ---- YES CONSTRAINT SET ---- # 
graph = reset_graph(use_constraints=True)
insert_query = graph.query("""
        WITH $target_urls AS target_urls_list
        UNWIND target_urls_list AS target_url
        MATCH (src:Page {url: $source_url})
        MERGE (tgt:Page {url: target_url})
        MERGE (src)-[:CONNECTED_TO]->(tgt)
    """, params={"source_url": 'https://en.wikipedia.org/wiki/Napoleon', "target_urls": target_url_list}) # --> SEGFAULT


# LONGER TARGET LIST 
target_list = ['https://www.worldcat.org/issn/0098-3500',
 'https://en.wikipedia.org/wiki/Assembly_language',
 'https://en.wikipedia.org/wiki/Sabbatical',
 'https://en.wikipedia.org/wiki/Kazushige_Goto',
 'https://en.wikipedia.org/wiki/Wikipedia:File_Upload_Wizard',
 'https://en.wikipedia.org/wiki/TFLOPS',
 'https://en.wikipedia.org/wiki/BSD_license',
 'https://en.wikipedia.org/wiki/Multiprocessing',
 'https://en.wikipedia.org/wiki/LAPACK']

graph = reset_graph(use_constraints=True)
insert_query = graph.query("""
        WITH $target_urls AS target_urls_list
        UNWIND target_urls_list AS target_url
        MATCH (src:Page {url: $source_url})
        MERGE (tgt:Page {url: target_url})
        MERGE (src)-[:CONNECTED_TO]->(tgt)
    """, params={"source_url": 'https://en.wikipedia.org/wiki/Napoleon', "target_urls": target_url_list})

ct = graph.query("MATCH (n:Page) RETURN COUNT(n)").result_set[0][0]
print(ct) # Returns 7??? 

url_match_set = graph.query("MATCH (n:Page) RETURN n.url").result_set
print(url_match_set) # Throws UnicodeDecodeError. Malformed URLs here.```

@alan-salimov-vai
Copy link

Seems like this is related to the index return from graph.list_indices().

@gkorland gkorland added the bug Something isn't working label Apr 30, 2024
@swilly22
Copy link
Contributor

Thank you @linuxfreakus, @alan-salimov-vai
We're able to replicate the issue from our end and are investigating.

@swilly22 swilly22 mentioned this issue May 1, 2024
Merged
@swilly22 swilly22 reopened this May 1, 2024
@swilly22 swilly22 linked a pull request May 1, 2024 that will close this issue
Ref count record #663
Merged
@swilly22
Copy link
Contributor

swilly22 commented May 2, 2024

Hi @linuxfreakus, @alan-salimov-vai we've been able to resolve both issues and are in progress of releasing these fixes.

@gkorland gkorland mentioned this issue May 13, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@swilly22 swilly22

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update Index, do not escape TAG field FalkorDB/FalkorDB
Ref count record FalkorDB/FalkorDB
4 participants
@gkorland @linuxfreakus @swilly22 @alan-salimov-vai