[CRASH] Redis 6.2.5 calling script - Apple M1 docker - ReJSON

[codekrafter]

Crash report

Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.

=== REDIS BUG REPORT START: Cut & paste starting from here ===

1:M 22 Jan 2022 21:37:12.022 # Redis 6.2.5 crashed by signal: 11, si_code: 1

1:M 22 Jan 2022 21:37:12.022 # Accessing address: 0x4000166a

1:M 22 Jan 2022 21:37:12.022 # Crashed running the instruction at: 0x400004b644


------ STACK TRACE ------

EIP:

/usr/local/bin/redis-server *:6379(dictSdsKeyCompare+0x34)[0x400004b644]


Backtrace:

/lib/x86_64-linux-gnu/libpthread.so.0(+0x14140)[0x4002158140]

/usr/local/bin/redis-server *:6379(dictSdsKeyCompare+0x34)[0x400004b644]

/usr/local/bin/redis-server *:6379(dictAddRaw+0x155)[0x4000049af5]

/usr/local/bin/redis-server *:6379(dictAdd+0x11)[0x4000049c81]

/usr/local/bin/redis-server *:6379(dbAdd+0x25)[0x400006fbe5]

/usr/local/bin/redis-server *:6379(hashTypeLookupWriteOrCreate+0x67)[0x4000092637]

/usr/local/bin/redis-server *:6379(hsetCommand+0x30)[0x4000092bf0]

/usr/local/bin/redis-server *:6379(call+0xa1)[0x400004f161]

/usr/local/bin/redis-server *:6379(luaRedisGenericCommand+0x389)[0x40000b5ef9]

/usr/local/bin/redis-server *:6379(+0x10d378)[0x400010d378]

/usr/local/bin/redis-server *:6379(+0x1175b0)[0x40001175b0]

/usr/local/bin/redis-server *:6379(+0x10da15)[0x400010da15]

/usr/local/bin/redis-server *:6379(+0x10cd48)[0x400010cd48]

/usr/local/bin/redis-server *:6379(+0x10dbc0)[0x400010dbc0]

/usr/local/bin/redis-server *:6379(lua_pcall+0x48)[0x400010b0a8]

/usr/local/bin/redis-server *:6379(evalGenericCommand+0x1f0)[0x40000b3870]

/usr/local/bin/redis-server *:6379(call+0xa1)[0x400004f161]

/usr/local/bin/redis-server *:6379(processCommand+0x5b3)[0x4000050d03]

/usr/local/bin/redis-server *:6379(processInputBuffer+0xf8)[0x4000063c28]

/usr/local/bin/redis-server *:6379(+0xfac98)[0x40000fac98]

/usr/local/bin/redis-server *:6379(aeProcessEvents+0x292)[0x4000047d82]

/usr/local/bin/redis-server *:6379(aeMain+0x1d)[0x4000047fed]

/usr/local/bin/redis-server *:6379(main+0x316)[0x4000044286]

/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x400218cd0a]

/usr/local/bin/redis-server *:6379(_start+0x2a)[0x400004475a]


------ REGISTERS ------

1:M 22 Jan 2022 21:37:12.030 # 

RAX:000000400004b6e8 RBX:000000400293c430

RCX:000000400019b220 RDX:0000000000000044

RDI:000000400629c283 RSI:400000004000166b

RBP:0000000000000000 RSP:0000004001c3f4c8

R8 :0000000000000000 R9 :00000040026000c0

R10:0000000000000005 R11:000000400293a000

R12:0000004001c3e399 R13:000000400293c450

R14:000000400293c420 R15:000000400629c283

RIP:000000400004b644 EFL:0000000000000207

CSGSFS:002b000000000033

1:M 22 Jan 2022 21:37:12.030 # (0000004001c3f4d7) -> 0000004002a35000

1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4d6) -> 00000040061f7cf0

1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4d5) -> 00000040061f7d50

1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4d4) -> 0000004000049c81

1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4d3) -> 0000004000209350

1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4d2) -> 00000012f1780fe9

1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4d1) -> 0000004002a42540

1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4d0) -> 00000040061f7d50

1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4cf) -> 00000040061f7cf0

1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4ce) -> 000000400293c420

1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4cd) -> 0000000000000001

1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4cc) -> 000000000000017b

1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4cb) -> 000000400293c470

1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4ca) -> 0000000000000bd8

1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4c9) -> 5f6518bc5329297b

1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4c8) -> 0000004000049af5


------ INFO OUTPUT ------

# Server

redis_version:6.2.5

redis_git_sha1:00000000

redis_git_dirty:0

redis_build_id:3afff201ac09d18c

redis_mode:standalone

os:Linux 5.10.76-linuxkit x86_64

arch_bits:64

multiplexing_api:epoll

atomicvar_api:atomic-builtin

gcc_version:10.2.1

process_id:1

process_supervised:no

run_id:9db7c01289f160a850ea0596de1a8c78dc7ea7ef

tcp_port:6379

server_time_usec:1642887432021178

uptime_in_seconds:104

uptime_in_days:0

hz:10

configured_hz:10

lru_clock:15497479

executable:/usr/local/bin/redis-server

config_file:

io_threads_active:0


# Clients

connected_clients:22

cluster_connections:0

maxclients:10000

client_recent_max_input_buffer:731

client_recent_max_output_buffer:0

blocked_clients:4

tracking_clients:0

clients_in_timeout_table:4


# Memory

used_memory:3790216

used_memory_human:3.61M

used_memory_rss:0

used_memory_rss_human:0B

used_memory_peak:4012296

used_memory_peak_human:3.83M

used_memory_peak_perc:94.47%

used_memory_overhead:1443336

used_memory_startup:904800

used_memory_dataset:2346880

used_memory_dataset_perc:81.34%

allocator_allocated:3801080

allocator_active:4284416

allocator_resident:7626752

total_system_memory:2085158912

total_system_memory_human:1.94G

used_memory_lua:65536

used_memory_lua_human:64.00K

used_memory_scripts:19728

used_memory_scripts_human:19.27K

number_of_cached_scripts:6

maxmemory:0

maxmemory_human:0B

maxmemory_policy:noeviction

allocator_frag_ratio:1.13

allocator_frag_bytes:483336

allocator_rss_ratio:1.78

allocator_rss_bytes:3342336

rss_overhead_ratio:0.00

rss_overhead_bytes:-7626752

mem_fragmentation_ratio:0.00

mem_fragmentation_bytes:-3729032

mem_not_counted_for_evict:0

mem_replication_backlog:0

mem_clients_slaves:0

mem_clients_normal:452512

mem_aof_buffer:0

mem_allocator:jemalloc-5.1.0

active_defrag_running:0

lazyfree_pending_objects:0

lazyfreed_objects:0


# Persistence

loading:0

current_cow_size:0

current_cow_size_age:0

current_fork_perc:0.00

current_save_keys_processed:0

current_save_keys_total:0

rdb_changes_since_last_save:502

rdb_bgsave_in_progress:0

rdb_last_save_time:1642887328

rdb_last_bgsave_status:ok

rdb_last_bgsave_time_sec:-1

rdb_current_bgsave_time_sec:-1

rdb_last_cow_size:0

aof_enabled:0

aof_rewrite_in_progress:0

aof_rewrite_scheduled:0

aof_last_rewrite_time_sec:-1

aof_current_rewrite_time_sec:-1

aof_last_bgrewrite_status:ok

aof_last_write_status:ok

aof_last_cow_size:0

module_fork_in_progress:0

module_fork_last_cow_size:0


# Stats

total_connections_received:22

total_commands_processed:1403

instantaneous_ops_per_sec:29

total_net_input_bytes:315081

total_net_output_bytes:563246

instantaneous_input_kbps:2.13

instantaneous_output_kbps:5.87

rejected_connections:0

sync_full:0

sync_partial_ok:0

sync_partial_err:0

expired_keys:0

expired_stale_perc:0.00

expired_time_cap_reached_count:0

expire_cycle_cpu_milliseconds:24

evicted_keys:0

keyspace_hits:1271

keyspace_misses:78

pubsub_channels:8

pubsub_patterns:0

latest_fork_usec:0

total_forks:0

migrate_cached_sockets:0

slave_expires_tracked_keys:0

active_defrag_hits:0

active_defrag_misses:0

active_defrag_key_hits:0

active_defrag_key_misses:0

tracking_total_keys:0

tracking_total_items:0

tracking_total_prefixes:0

unexpected_error_replies:0

total_error_replies:41

dump_payload_sanitizations:0

total_reads_processed:387

total_writes_processed:764

io_threaded_reads_processed:0

io_threaded_writes_processed:0


# Replication

role:master

connected_slaves:0

master_failover_state:no-failover

master_replid:a740067518cdb3377a576b9b36354c391a907c9a

master_replid2:0000000000000000000000000000000000000000

master_repl_offset:0

second_repl_offset:-1

repl_backlog_active:0

repl_backlog_size:1048576

repl_backlog_first_byte_offset:0

repl_backlog_histlen:0


# CPU

used_cpu_sys:0.842743

used_cpu_user:1.708462

used_cpu_sys_children:0.011343

used_cpu_user_children:0.049196

used_cpu_sys_main_thread:0.832938

used_cpu_user_main_thread:1.685633


# Modules

module:name=ReJSON,ver=20004,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]

module:name=search,ver=20205,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]


# Commandstats

cmdstat_get:calls=22,usec=234,usec_per_call=10.64,rejected_calls=0,failed_calls=0

cmdstat_rpoplpush:calls=43,usec=207,usec_per_call=4.81,rejected_calls=0,failed_calls=0

cmdstat_expire:calls=1,usec=249,usec_per_call=249.00,rejected_calls=0,failed_calls=0

cmdstat_monitor:calls=1,usec=126,usec_per_call=126.00,rejected_calls=0,failed_calls=0

cmdstat_zrangebyscore:calls=57,usec=5103,usec_per_call=89.53,rejected_calls=0,failed_calls=0

cmdstat_evalsha:calls=157,usec=244241,usec_per_call=1555.68,rejected_calls=0,failed_calls=20

cmdstat_zscore:calls=22,usec=1428,usec_per_call=64.91,rejected_calls=0,failed_calls=0

cmdstat_multi:calls=22,usec=86,usec_per_call=3.91,rejected_calls=0,failed_calls=0

cmdstat_zrem:calls=44,usec=1538,usec_per_call=34.95,rejected_calls=0,failed_calls=0

cmdstat_del:calls=22,usec=484,usec_per_call=22.00,rejected_calls=0,failed_calls=0

cmdstat_exists:calls=80,usec=555,usec_per_call=6.94,rejected_calls=0,failed_calls=0

cmdstat_lrem:calls=44,usec=1272,usec_per_call=28.91,rejected_calls=0,failed_calls=0

cmdstat_auth:calls=21,usec=334,usec_per_call=15.90,rejected_calls=0,failed_calls=21

cmdstat_hmset:calls=44,usec=1688,usec_per_call=38.36,rejected_calls=0,failed_calls=0

cmdstat_command:calls=1,usec=10173,usec_per_call=10173.00,rejected_calls=0,failed_calls=0

cmdstat_hset:calls=45,usec=1409,usec_per_call=31.31,rejected_calls=0,failed_calls=0

cmdstat_keys:calls=1,usec=366,usec_per_call=366.00,rejected_calls=0,failed_calls=0

cmdstat_smembers:calls=4,usec=63,usec_per_call=15.75,rejected_calls=0,failed_calls=0

cmdstat_info:calls=14,usec=7414,usec_per_call=529.57,rejected_calls=0,failed_calls=0

cmdstat_json.set:calls=68,usec=21506,usec_per_call=316.26,rejected_calls=0,failed_calls=0

cmdstat_zrange:calls=78,usec=3810,usec_per_call=48.85,rejected_calls=0,failed_calls=0

cmdstat_zadd:calls=65,usec=4267,usec_per_call=65.65,rejected_calls=0,failed_calls=0

cmdstat_lpush:calls=23,usec=1594,usec_per_call=69.30,rejected_calls=0,failed_calls=0

cmdstat_publish:calls=224,usec=2361,usec_per_call=10.54,rejected_calls=0,failed_calls=0

cmdstat_lrange:calls=4,usec=40,usec_per_call=10.00,rejected_calls=0,failed_calls=0

cmdstat_hget:calls=23,usec=538,usec_per_call=23.39,rejected_calls=0,failed_calls=0

cmdstat_exec:calls=22,usec=4246,usec_per_call=193.00,rejected_calls=0,failed_calls=0

cmdstat_hgetall:calls=65,usec=1886,usec_per_call=29.02,rejected_calls=0,failed_calls=0

cmdstat_srem:calls=44,usec=319,usec_per_call=7.25,rejected_calls=0,failed_calls=0

cmdstat_brpoplpush:calls=35,usec=6332,usec_per_call=180.91,rejected_calls=0,failed_calls=0

cmdstat_client:calls=26,usec=228,usec_per_call=8.77,rejected_calls=0,failed_calls=0

cmdstat_set:calls=30,usec=3504,usec_per_call=116.80,rejected_calls=0,failed_calls=0

cmdstat_subscribe:calls=8,usec=378,usec_per_call=47.25,rejected_calls=0,failed_calls=0

cmdstat_eval:calls=20,usec=86805,usec_per_call=4340.25,rejected_calls=0,failed_calls=0

cmdstat_incr:calls=23,usec=787,usec_per_call=34.22,rejected_calls=0,failed_calls=0


# Errorstats

errorstat_ERR:count=21

errorstat_NOSCRIPT:count=20


# Cluster

cluster_enabled:0


# Keyspace

db0:keys=1024,expires=21,avg_ttl=24087363


------ CLIENT LIST OUTPUT ------

id=9 addr=172.17.0.1:59028 laddr=172.17.0.3:6379 fd=10 name=local:bull:Y2Fk age=20 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=729 qbuf-free=40225 argv-mem=589 obl=0 oll=0 omem=0 tot-mem=62189 events=r cmd=evalsha user=default redir=-1

id=10 addr=172.17.0.1:59030 laddr=172.17.0.3:6379 fd=11 name=local:bull:Y2FkLWNhY2hl age=20 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20584 events=r cmd=evalsha user=default redir=-1

id=11 addr=172.17.0.1:59032 laddr=172.17.0.3:6379 fd=12 name=local:bull:ZGlzY29yZA== age=20 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20584 events=r cmd=evalsha user=default redir=-1

id=12 addr=172.17.0.1:59034 laddr=172.17.0.3:6379 fd=13 name= age=20 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=publish user=default redir=-1

id=13 addr=172.17.0.1:59026 laddr=172.17.0.3:6379 fd=14 name=local:bull:Y2Fk age=20 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=hgetall user=default redir=-1

id=14 addr=172.17.0.1:59038 laddr=172.17.0.3:6379 fd=15 name= age=20 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=info user=default redir=-1

id=15 addr=172.17.0.1:59036 laddr=172.17.0.3:6379 fd=16 name=local:bull:cGF0cm9scw== age=20 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20584 events=r cmd=evalsha user=default redir=-1

id=7 addr=172.17.0.1:59022 laddr=172.17.0.3:6379 fd=8 name= age=30 idle=0 flags=O db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=879 oll=0 omem=0 tot-mem=20496 events=r cmd=monitor user=default redir=-1

id=8 addr=172.17.0.1:59024 laddr=172.17.0.3:6379 fd=9 name=local:bull:ZGlzY29yZC1ub3RpZg== age=20 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=info user=default redir=-1

id=16 addr=172.17.0.1:59044 laddr=172.17.0.3:6379 fd=17 name= age=19 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=info user=default redir=-1

id=17 addr=172.17.0.1:59046 laddr=172.17.0.3:6379 fd=18 name= age=19 idle=15 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1

id=18 addr=172.17.0.1:59050 laddr=172.17.0.3:6379 fd=19 name= age=19 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=info user=default redir=-1

id=19 addr=172.17.0.1:59052 laddr=172.17.0.3:6379 fd=20 name= age=19 idle=19 flags=P db=0 sub=4 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1

id=25 addr=172.17.0.1:59064 laddr=172.17.0.3:6379 fd=26 name=local:bull:cGF0cm9scw== age=19 idle=4 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=59 obl=0 oll=0 omem=0 tot-mem=20579 events=r cmd=brpoplpush user=default redir=-1

id=26 addr=172.17.0.1:59066 laddr=172.17.0.3:6379 fd=27 name=local:bull:ZGlzY29yZA== age=19 idle=4 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=540 qbuf-free=0 argv-mem=59 obl=0 oll=0 omem=0 tot-mem=21211 events=r cmd=brpoplpush user=default redir=-1

id=27 addr=172.17.0.1:59068 laddr=172.17.0.3:6379 fd=28 name=local:bull:Y2FkLWNhY2hl age=19 idle=4 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=94 qbuf-free=0 argv-mem=63 obl=0 oll=0 omem=0 tot-mem=20687 events=r cmd=brpoplpush user=default redir=-1

id=28 addr=172.17.0.1:59070 laddr=172.17.0.3:6379 fd=29 name=local:bull:Y2Fk age=19 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=82 argv-mem=51 obl=0 oll=0 omem=0 tot-mem=20659 events=r cmd=brpoplpush user=default redir=-1

id=20 addr=172.17.0.1:59048 laddr=172.17.0.3:6379 fd=21 name= age=19 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1

id=21 addr=172.17.0.1:59056 laddr=172.17.0.3:6379 fd=22 name=local:bull:Y2FkLWNhY2hl age=19 idle=19 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1

id=22 addr=172.17.0.1:59058 laddr=172.17.0.3:6379 fd=23 name=local:bull:Y2Fk age=19 idle=0 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1

id=23 addr=172.17.0.1:59062 laddr=172.17.0.3:6379 fd=24 name=local:bull:ZGlzY29yZA== age=19 idle=19 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1

id=24 addr=172.17.0.1:59060 laddr=172.17.0.3:6379 fd=25 name=local:bull:cGF0cm9scw== age=19 idle=4 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1


------ CURRENT CLIENT INFO ------

id=9 addr=172.17.0.1:59028 laddr=172.17.0.3:6379 fd=10 name=local:bull:Y2Fk age=20 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=729 qbuf-free=40225 argv-mem=589 obl=0 oll=0 omem=0 tot-mem=62189 events=r cmd=evalsha user=default redir=-1

argv[0]: 'evalsha'

argv[1]: '8f55ae4a3be429c6d38c5d5db3e80edf89197b64'

argv[2]: '6'

argv[3]: 'local:bull:cad:wait'

argv[4]: 'local:bull:cad:paused'

argv[5]: 'local:bull:cad:meta-paused'

argv[6]: 'local:bull:cad:id'

argv[7]: 'local:bull:cad:delayed'

argv[8]: 'local:bull:cad:priority'

argv[9]: 'local:bull:cad:'

argv[10]: 'repeat:757805924c3d6f0c532f0e36651e4769:1642887433000'

argv[11]: 'warn inactivity'

argv[12]: '{}'

argv[13]: '{"repeat":{"count":970,"key":"warn inactivity:inactivity-warning::1000","every":1000,"jobId":"inactivity-warning"},"jobId":"repeat:757805924c3d6f0c532f0e36651e4769:1642887433000","delay":980,"timestamp":1642887432020,"prevMillis":1642887433000,"attempts":1}'

argv[14]: '1642887432020'

argv[15]: '980'

argv[16]: '1642887433000'

argv[17]: '0'

argv[18]: 'LPUSH'

argv[19]: 'e9288e3f-961b-4a08-a6aa-19fb873b5ab2'


------ MODULES INFO OUTPUT ------

# ReJSON_trace

ReJSON_trace:   0: redis_module::base_info_func

   1: modulesCollectInfo

             at /usr/src/redis/src/module.c:7076:9

   2: logModulesInfo

             at /usr/src/redis/src/debug.c:1598:22

   3: printCrashReport

             at /usr/src/redis/src/debug.c:1849:5

      sigsegvHandler

             at /usr/src/redis/src/debug.c:1831:5

   4: <unknown>

   5: dictSdsKeyCompare

             at /usr/src/redis/src/server.c:1259:8

   6: _dictKeyIndex

             at /usr/src/redis/src/dict.c:1038:33

      dictAddRaw

             at /usr/src/redis/src/dict.c:325:18

   7: dictAdd

             at /usr/src/redis/src/dict.c:290:24

   8: dbAdd

             at /usr/src/redis/src/db.c:194:18

   9: hashTypeLookupWriteOrCreate

             at /usr/src/redis/src/t_hash.c:454:9

      hashTypeLookupWriteOrCreate

             at /usr/src/redis/src/t_hash.c:448:7

  10: hsetCommand

             at /usr/src/redis/src/t_hash.c:663:14

  11: call

             at /usr/src/redis/src/server.c:3717:5

  12: luaRedisGenericCommand

             at /usr/src/redis/src/scripting.c:736:5

  13: luaD_precall

  14: luaV_execute

  15: luaD_call

  16: luaD_rawrunprotected

  17: luaD_pcall

  18: lua_pcall

  19: evalGenericCommand

             at /usr/src/redis/src/scripting.c:1598:11

  20: call

             at /usr/src/redis/src/server.c:3717:5

  21: processCommand

             at /usr/src/redis/src/server.c:4242:9

  22: processCommandAndResetClient

             at /usr/src/redis/src/networking.c:2018:9

      processInputBuffer

             at /usr/src/redis/src/networking.c:2119:17

  23: callHandler

             at /usr/src/redis/src/connhelpers.h:79:18

      connSocketEventHandler

             at /usr/src/redis/src/connection.c:295:14

  24: aeProcessEvents

             at /usr/src/redis/src/ae.c:427:17

  25: aeMain

             at /usr/src/redis/src/ae.c:487:9

  26: main

             at /usr/src/redis/src/server.c:6396:5

  27: __libc_start_main

  28: _start



------ FAST MEMORY TEST ------

1:M 22 Jan 2022 21:37:12.094 # Bio thread for job type #0 terminated

1:M 22 Jan 2022 21:37:12.095 # Bio thread for job type #1 terminated

1:M 22 Jan 2022 21:37:12.095 # Bio thread for job type #2 terminated

*** Preparing to test memory region 4000213000 (2682880 bytes)

*** Preparing to test memory region 4001c6c000 (12288 bytes)

*** Preparing to test memory region 4001db5000 (8192 bytes)

*** Preparing to test memory region 4002140000 (16384 bytes)

*** Preparing to test memory region 4002162000 (16384 bytes)

*** Preparing to test memory region 4002327000 (40960 bytes)

*** Preparing to test memory region 4002600000 (2097152 bytes)

*** Preparing to test memory region 4002931000 (2097152 bytes)

*** Preparing to test memory region 4003000000 (4194304 bytes)

*** Preparing to test memory region 4003938000 (12288 bytes)

*** Preparing to test memory region 400393c000 (8388608 bytes)

*** Preparing to test memory region 400413d000 (8388608 bytes)

*** Preparing to test memory region 400493e000 (8388608 bytes)

*** Preparing to test memory region 400513f000 (8388608 bytes)

*** Preparing to test memory region 4005940000 (8388608 bytes)

*** Preparing to test memory region 4006140000 (5767168 bytes)

*** Preparing to test memory region 400740b000 (3670016 bytes)

.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O

Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.


------ DUMPING CODE AROUND EIP ------

Symbol: dictSdsKeyCompare (base: 0x400004b610)

Module: /usr/local/bin/redis-server *:6379 (base 0x4000000000)

$ xxd -r -p /tmp/dump.hex /tmp/dump.bin

$ objdump --adjust-vma=0x400004b610 -D -b binary -m i386:x86-64 /tmp/dump.bin

------

1:M 22 Jan 2022 21:37:12.394 # dump of function (hexdump of 180 bytes):

4889f74889d60fb657ff89d083e0073c040f879f7bffff488d0df2fb14000fb6c0486304814801c8ffe0660f1f440000486357f70fb64eff89c883e0073c040f876a7bffff4c8d05d8fb14000fb6c0496304804c01c0ffe00f1f8400000000008b4ef731c039ca75214883ec08e8ee6cffff85c00f94c04883c4080fb6c0c3900fb74efb31c039ca74dfc30f1f4400000fb64efdebcd662e0f1f840000000000c0e9030fb6c9ebbb0f1f8400000000008b4eefeb


=== REDIS BUG REPORT END. Make sure to include from START to END. ===

Additional information

This from running a stock ReJSON docker container, with no other modules. Most of the redis interaction is from the Bull queuing library

[oranagra]

@MeirShpilraien can you please take a look?

[MeirShpilraien]

@codekrafter any chance you can share the script with this sha: 8f55ae4a3be429c6d38c5d5db3e80edf89197b64? Can help reproduce the issue.

[MeirShpilraien]

Also notice that RediSearch and RedisJson is also loaded, we need to also considered that, might be a module that causes memory corruption ...

[codekrafter]

@MeirShpilraien it should be this script: https://github.com/OptimalBits/bull/blob/v4.2.1/lib/commands/addJob-6.lua
This is a crash from my local development environment (an M1 mac if that is of interest), but I know my production redis instance has been crashing every so often, I haven't gotten a chance to pull the logs from that yet but I would bet it is the same problem.

[PingXie]

Well the crash log you posted is about a Redis server running on x64 Linux.

Based on the instruction where this crash occurred, the invalid memory pointer belongs to a key string that is already in the database. Assuming the physical memory module in your system is not flaky, this sounds like a use-after-free issue to me. If it reproduces relatively consistently on your local dev env, maybe try a few more repros with Valgrind?

0: 48 89 f7 mov rdi,rsi
3: 48 89 d6 mov rsi,rdx
6: 0f b6 57 ff movzx edx,BYTE PTR [rdi-0x1]
a: 89 d0 mov eax,edx
c: 83 e0 07 and eax,0x7
f: 3c 04 cmp al,0x4
11: 0f 87 9f 7b ff ff ja 0xffffffffffff7bb6
17: 48 8d 0d f2 fb 14 00 lea rcx,[rip+0x14fbf2] # 0x14fc10
1e: 0f b6 c0 movzx eax,al
21: 48 63 04 81 movsxd rax,DWORD PTR [rcx+rax4]
25: 48 01 c8 add rax,rcx
28: ff e0 jmp rax
2a: 66 0f 1f 44 00 00 nop WORD PTR [rax+rax1+0x0]
30: 48 63 57 f7 movsxd rdx,DWORD PTR [rdi-0x9]
34: 0f b6 4e ff movzx ecx,BYTE PTR [rsi-0x1] // <--- here %rsi is key2 in dictSdsKeyCompare,

%rsi = 0x4000166b

static int dictSdsKeyCompare(void *privdata, const void *key1, const void *key2)

which is invoked by

static long _dictKeyIndex(dict *d, const void *key, uint64_t hash, dictEntry **existing) {
...
if (key==he->key || dictCompareKeys(d, key, he->key)) { // <-- here he->key is key2
...
}

[MeirShpilraien]

@codekrafter I do not see anything suspicious in the Lua script. I suspects its the modules fault. Is it possible to run without the module (just to try to prove/disprove this theory)?

[codekrafter]

@MeirShpilraien most of my app's interaction with redis is through module commands, which makes it hard to test the application without the module.

[MeirShpilraien]

@codekrafter any chance you have a deterministic reproduction steps?

[sundb]

@codekrafter It looks like this crash is similar to #10341.
If you are running docker on M1, can you confirm if you are using qemu?
What is the redis docker image you are using?

[codekrafter]

@sundb How would I confirm I am using qemu? I believe so as my laptop is m1 but it is running as x86. It is using the redislabs/rejson container.

[sundb]

@codekrafter You can tell by the warning shown in the docker.
ref comment: #10341 (comment)

[codekrafter]

@sundb can confirm it is running via qemu

[sundb]

@codekrafter It seems to be necessary to make an issue with https://github.com/RedisJSON/RedisJSON and ask them to build an image that supports arm, otherwise it will definitely have problems running on m1 docker.

[oranagra]

i'm closing all of these reports, should be tracked in the ReJson repo.
@sundb are all these reports about Arm / docker issues coming from the JSON / module dockers? or any of them are plain redis?

[sundb]

@oranagra RedisJson and RedisSearch, no plain redis.
#9561 plain redis
#10341 plain redis
#9187 RedisGraph
#10162 RedisSearch
#10442 RedisJson

Here are the available ARM docker image:

plain redis: hub.docker.com/_/redis
modules: hub.docker.com/r/redis/redis-stack-server

The above version may not be the latest, someon can find it on the docker hub if needed.

[oranagra]

I see you mentioned that ReJson doesn't have an ARM image, does redis have one?
if we can't fix this by creating a docker image, i think we need to solve this by preventing redis from running (these reports are starting to be annoying).

I think we should find a way to detect this issue, and abort redis on startup with a clear message. like we did in #8224.
i suppose that being under qemu, we don't have a good way to detect that combination, but maybe we can just do a calloc() and check if the memory is zeroed?
can someone with spare time and an M1 machine test this?

[sundb]

@oranagra I modified the above comment to add the available images.
I remembered a silly thing on the bike ride, jemalloc already gives us a way to detect memory not zero in https://github.com/jemalloc/jemalloc/pull/2005/files, we can actually use memset in zcalloc when it is detected.
Wait for me to make a pr.

[oranagra]

@sundb i don't think it's wise to add code to memset on calloc, if that's what you meant.
first, IIUC, the problem is not with any calloc, just when re-using an page that was previously dismissed, so doing that would mean that in some cases we zero twice, and in other cases we're missing out on lazy zeroing.
secondly, even adding an additional branch in our low level function, just for better support of a platform that's not a realistic target for redis, would be wasteful.

Another alternative is to just enable the feature added in the jemalloc PR you mentioned (which causes jemalloc to never dismiss pages when running on an affected platform), that would fix the problem on qemu, by adding a sub-optimal effect on the runtime.

what i suggest is to copy that detection code from jemalloc into redis, and use it in a similar manner as we did in #8224.
If the user disables the warning, we can then enable that feature (disable trust_madvise)

[sundb]

@oranagra I found some problems when I did with the way you said.

1. When the MADV_DONTNEED zeroes page failure is detected (it must be placed after the config loaded because we need to log something), it is actually too late, and jemalloc may have allocte some memory before this.
2. trust_madvis is read-only, I'm not sure jemalloc is trying to prevent us from setting it dynamically, as in the question in 1).
   Please correct me if I'm wrong, as I'm not familiar with jemalloc.

[oranagra]

@sundb i don't want to invest too much effort (and code) in this, if it's too late to tweak jemalloc config, let's drop this idea.
if we do this detection, logging and exit after we already allocated things, there's also a slim chance that some calloc already failed zeroing memory (but that's actually unlikely, since it's unlikely that we used MADV_DONTNEED).

i think the most important requirements here are:

1. avoid letting redis start and later crash at runtime (after being populated with data)
2. avoid people opening more issues about odd crashes.

from that perspective, i think it's fine to do that check after init, and forget about the idea of setting trust_madvise to 0 when the warning (exit()) was silenced.

in theory, people can provide jemalloc with a config from outside (there's a mechanism for that), and set trust_madvise. in theory we can detect that it has been disabled, and silence our warning too, but i guess i don't mind requiring these people to also silence the warning in redis config file separately.

[oranagra]

@sundb i understand some progress have been made, and since we posted quite a few links to this comment from other pages and people are likely to hit it, i'd like to ask you to edit / update it.

it currently refers to redisfab which i understand is a testing playground, and not something seriously maintained.
instead, the official multi-platform docker images we should direct people to are:

• plain redis: https://hub.docker.com/_/redis
• modules: https://hub.docker.com/r/redis/redis-stack-server

[sundb]

@oranagra Already updated.