Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consistent crash with Redis 6.2.4 #1124

Closed
nihohit opened this issue Oct 26, 2023 · 3 comments
Closed

Consistent crash with Redis 6.2.4 #1124

nihohit opened this issue Oct 26, 2023 · 3 comments

Comments

@nihohit
Copy link

nihohit commented Oct 26, 2023

Redis 6.2.4 crashes consistently when loaded with a release build of the current (90ba302) commit. This happens before the server finishes loading, with the command redis-server --loadmodule /home/ubuntu/repositories/RedisJSON/target/release/librejson.so --logfile /tmp/redisgMYd0p/redis.log --port 40505 --bind 127.0.0.1.

OS :
Ubuntu 22.0

architecture: x86-64

logs:

2010285:C 26 Oct 2023 08:55:39.358 # Redis version=6.2.4, bits=64, commit=00000000, modified=0, pid=2010285, just started
2010285:C 26 Oct 2023 08:55:39.358 # Configuration loaded
2010285:M 26 Oct 2023 08:55:39.359 * monotonic clock: POSIX clock_gettime
2010285:M 26 Oct 2023 08:55:39.359 * Running mode=standalone, port=40505.
2010285:M 26 Oct 2023 08:55:39.359 # Server initialized
2010285:M 26 Oct 2023 08:55:39.359 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2010285:M 26 Oct 2023 08:55:39.360 * <ReJSON> Created new data type 'ReJSON-RL'


=== REDIS BUG REPORT START: Cut & paste starting from here ===
2010285:M 26 Oct 2023 08:55:39.461 # Redis 6.2.4 crashed by signal: 6, si_code: -6
2010285:M 26 Oct 2023 08:55:39.461 # Killed by PID: 2010285, UID: 1000
2010285:M 26 Oct 2023 08:55:39.461 # Crashed running the instruction at: 0x7fbd3cc969fc

------ STACK TRACE ------
EIP:
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7fbd3cc969fc]

Backtrace:
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fbd3cc42520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7fbd3cc969fc]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7fbd3cc42476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7fbd3cc287f3]
/home/ubuntu/repositories/RedisJSON/target/release/librejson.so(+0x1fabb7)[0x7fbd3bdfabb7]
/home/ubuntu/repositories/RedisJSON/target/release/librejson.so(+0x1f8ca4)[0x7fbd3bdf8ca4]
/home/ubuntu/repositories/RedisJSON/target/release/librejson.so(+0x1f8bb2)[0x7fbd3bdf8bb2]
/home/ubuntu/repositories/RedisJSON/target/release/librejson.so(+0x1f88b1)[0x7fbd3bdf88b1]
/home/ubuntu/repositories/RedisJSON/target/release/librejson.so(+0x1f79f6)[0x7fbd3bdf79f6]
/home/ubuntu/repositories/RedisJSON/target/release/librejson.so(+0x1f8642)[0x7fbd3bdf8642]
/home/ubuntu/repositories/RedisJSON/target/release/librejson.so(+0x4bda3)[0x7fbd3bc4bda3]
/home/ubuntu/repositories/RedisJSON/target/release/librejson.so(+0x4be33)[0x7fbd3bc4be33]
/home/ubuntu/repositories/RedisJSON/target/release/librejson.so(+0xe2b67)[0x7fbd3bce2b67]
/home/ubuntu/repositories/RedisJSON/target/release/librejson.so(RedisModule_OnLoad+0x16b8)[0x7fbd3bc96b48]
redis-server 127.0.0.1:40505(moduleLoad+0xc0)[0x556e4d962c40]
redis-server 127.0.0.1:40505(moduleLoadFromQueue+0x43)[0x556e4d962e53]
redis-server 127.0.0.1:40505(main+0x2ae)[0x556e4d8c03ee]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fbd3cc29d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fbd3cc29e40]
redis-server 127.0.0.1:40505(_start+0x25)[0x556e4d8c0975]

------ REGISTERS ------
2010285:M 26 Oct 2023 08:55:39.462 # 
RAX:0000000000000000 RBX:00007fbd3d597f80
RCX:00007fbd3cc969fc RDX:0000000000000006
RDI:00000000001eacad RSI:00000000001eacad
RBP:00000000001eacad RSP:00007ffc1c823ad0
R8 :00007ffc1c823ba0 R9 :0000000000000001
R10:0000000000000008 R11:0000000000000246
R12:0000000000000006 R13:0000000000000016
R14:00007fbd3bf072a0 R15:00007ffc1c823e00
RIP:00007fbd3cc969fc EFL:0000000000000246
CSGSFS:002b000000000033
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823adf) -> 0000000000000000
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823ade) -> 0000000000000000
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823add) -> 0000000000000000
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823adc) -> 0000000000000000
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823adb) -> 0000000000000000
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823ada) -> 0000000000000000
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823ad9) -> 0000000000000000
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823ad8) -> 0000000000000000
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823ad7) -> 0000000000000000
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823ad6) -> 0000000000000000
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823ad5) -> 0000000000000000
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823ad4) -> 4000000000000000
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823ad3) -> 0000556e4d8c0950
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823ad2) -> 0000000000000000
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823ad1) -> 0000000000000000
2010285:M 26 Oct 2023 08:55:39.462 # (00007ffc1c823ad0) -> 0000000000000000

------ INFO OUTPUT ------
# Server
redis_version:6.2.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:c14a6ae8d26280c2
redis_mode:standalone
os:Linux 6.2.0-1011-aws x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:11.4.0
process_id:2010285
process_supervised:no
run_id:3254ba61b866cb5899d9570f0d557fb1db5300a3
tcp_port:40505
server_time_usec:1698310539357351
uptime_in_seconds:0
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:3811723
executable:/home/ubuntu/repositories/backup/babushka/submodules/redis-rs/redis/redis-server
config_file:
io_threads_active:0

# Clients
connected_clients:0
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

# Memory
used_memory:55023456
used_memory_human:52.47M
used_memory_rss:0
used_memory_rss_human:0B
used_memory_peak:55023456
used_memory_peak_human:52.47M
used_memory_peak_perc:inf%
used_memory_overhead:0
used_memory_startup:0
used_memory_dataset:55023456
used_memory_dataset_perc:100.00%
allocator_allocated:0
allocator_active:0
allocator_resident:0
total_system_memory:32805904384
total_system_memory_human:30.55G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:-nan
allocator_frag_bytes:0
allocator_rss_ratio:-nan
allocator_rss_bytes:0
rss_overhead_ratio:-nan
rss_overhead_bytes:0
mem_fragmentation_ratio:-nan
mem_fragmentation_bytes:0
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1698310539
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:0
total_commands_processed:0
instantaneous_ops_per_sec:0
total_net_input_bytes:0
total_net_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:0
total_writes_processed:0
io_threaded_reads_processed:0
io_threaded_writes_processed:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:dbc3e50ef033952087627dbda878357e6050b83a
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.024731
used_cpu_user:0.082438
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.024731
used_cpu_user_main_thread:0.082438

# Modules

# Commandstats

# Errorstats

# Cluster
cluster_enabled:0

# Keyspace

------ CLIENT LIST OUTPUT ------

------ MODULES INFO OUTPUT ------

------ FAST MEMORY TEST ------
*** Preparing to test memory region 556e4da9f000 (2281472 bytes)
*** Preparing to test memory region 556e4e37a000 (270336 bytes)
*** Preparing to test memory region 7fbd34d00000 (38797312 bytes)
*** Preparing to test memory region 7fbd37c00000 (6291456 bytes)
*** Preparing to test memory region 7fbd38c00000 (18874368 bytes)
*** Preparing to test memory region 7fbd3c400000 (8388608 bytes)
*** Preparing to test memory region 7fbd3ce1b000 (53248 bytes)
*** Preparing to test memory region 7fbd3d440000 (12288 bytes)
*** Preparing to test memory region 7fbd3d596000 (24576 bytes)
*** Preparing to test memory region 7fbd3d730000 (8192 bytes)
.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: pthread_kill (base: 0x7fbd3cc968d0)
Module: /lib/x86_64-linux-gnu/libc.so.6 (base 0x7fbd3cc00000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x7fbd3cc968d0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
2010285:M 26 Oct 2023 08:55:39.626 # dump of function (hexdump of 428 bytes):
f30f1efa41568d56e0415541bd16000000415455534881ec9000000064488b042528000000488984248800000031c083fa01767e4889fb4189f464483b3c25100000000f84c70000004989e641ba0800000031ffb80e0000004c89f2488d352dc113000f0531c0488dab74090000ba01000000f00fb155000f85ca00000080bb730900000074594531ed31d287937409000083fa010f8fbd00000041ba0800000031d24c89f6bf02000000b80e0000000f05488b84248800000064482b0425280000000f859c0000004881c4900000004489e85b5d415c415d415ec30f1f4000448babd0020000e8845605004489e289c74489eeb8ea0000000f053d00f0ffff76854189c541f7ddeb80660f1f440000b8ba0000000f0589c5e8525605004489e289ee89c7b8ea0000000f054189c541f7dd3d00f0ffffb800000000440f46e8e96dffffff0f1f004889efe810a8ffffe929ffffff0f1f004889efe8d0a8ffffe936ffffffe806fd0900660f1f440000f30f1efac3662e0f1f840000000000904157415641554154554889fd534883ec2864488b042528000000488944241831c0648b0425d0020000894424

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash by opening an issue on github:

           http://github.com/redis/redis/issues

  Suspect RAM error? Use redis-server --test-memory to verify it.
nihohit added a commit to amazon-contributing/redis-rs that referenced this issue Oct 26, 2023
@LiorKogan
Copy link
Member

Since v2.6 GA the minimal supported Redis version is 7.2

nihohit added a commit to amazon-contributing/redis-rs that referenced this issue Oct 26, 2023
nihohit added a commit to amazon-contributing/redis-rs that referenced this issue Oct 26, 2023
nihohit added a commit to redis-rs/redis-rs that referenced this issue Oct 31, 2023
@nihohit nihohit closed this as completed Oct 31, 2023
barshaul pushed a commit to amazon-contributing/redis-rs that referenced this issue Dec 7, 2023
* Disable JSON module tests for redis 6.2.4. (redis-rs#980)

This happens due to a consistent crash -
RedisJSON/RedisJSON#1124

* Implement `FromRedisValue` for `Box<[T]>` and `Arc<[T]>` (redis-rs#799)

* Update minimal rust version to 1.6.5 (redis-rs#982)

It's required by regex v1.10.2.
https://github.com/redis-rs/redis-rs/actions/runs/6760188489/job/18373652560?pr=981

* Support Mutual TLS (redis-rs#858)

Add support for mutual TLS. Enable specifying client
certificate and key and (optional) root certificate.

* changing timeouts from usize and isize to f64 (redis-rs#988)

Co-authored-by: Eythor Helgason <eytorhelgason@gmail.com>

* Accept iterator at `ClusterClient` initialization (redis-rs#987)

* Prefer routing to primary in a transaction. (redis-rs#986)

* Rename route_pipeline to route_for_pipeline.

There are 2 `route_pipeline` functions in the file, so we change one to
avoid confusion.

* Prefer routing to primary in a transaction.

* CrossSlot error on pipeline with many routes.

* Sync Pub/Sub - cache received pub/sub messages. (redis-rs#910)

When sending subscibre/unsubscribe messages to the server, the pub/sub
client might receive messages that it can mistake to be the response,
and not return to the next `get_message` call. This fix ensures that
pub/sub messages will be queued until the next `get_message` call, so
that no message will be missed.

* Fix sync cluster behavior with transactions. (redis-rs#983)

The leading MULTI prevents the transaction from being properly routed.
In order to route the request correctly, we drop the leading MULTI from
the routable.

* Release redis 0.23.4 / redis-test 0.2.4 (redis-rs#992)

* Tests: Add retries to test cluster creation (redis-rs#994)

* Revert redis-test versioning changes (redis-rs#993)

Publishing to crates.io without the version specification
is currently not possible. Reverting these changes for now
until we have a better solution/workaround.

* Fix features for `load_native_certs`. (redis-rs#996)

The directive now matches the directives wrapping `load_native_certs`'s
`use` statements. This would fail compilation for anyone accidentally
using the "tls" feature with rustls.

* Bump aHash to v0.8.6 (redis-rs#966)

* Update Command expiration values to be an appropriate type (redis-rs#589)

This change updates expiration values provided to some commands in the
`Command` trait to be of type `i64` or `u64` where appropriate,
replacing the previously used `usize` type.

More specifically, the commands that are updated are

- SETEX, PSETEX
  - Expiration value changed from `usize` to `u64`
- EXPIRE, PEXPIRE, EXPIREAT, PEXPIREAT
  - Expiration value changed from `usize` to `i64`

For the `*SETEX` commands, there is no mention of supporting negative
values. And indeed, providing a negative value through redis-cli returns
an error message.

For the `*EXPIRE*` commands, I couldn't find the paragraph in the docs
referenced in redis-rs#575, nor any other mention that the commands accept a
negative expiration argument. However, in testing with redis-cli it
seems a negative expiration value is valid and in checking the code I
found it to confirm these observations. See [here][code1] and
[here][code2].

Note that this is a breaking change.

[code1]: https://github.com/redis/redis/blob/e3ef73dc2a557232c60c732705e8e6ff2050eba9/src/expire.c#L570-L571
[code2]: https://github.com/redis/redis/blob/e3ef73dc2a557232c60c732705e8e6ff2050eba9/src/expire.c#L483-L491

* Fix StreamId::contains_key signature (redis-rs#783)

* Order in usage of ClusterParams. (redis-rs#997)

* Move `tls_params` into `ClusterParams`.

The two objects are used together, so it makes sense to join them.

* Cluster: Contain `ClusterParams` internally.

This matches the async cluster's structure, and ensures that fields that
are added to `ClusterParams` are automatically added to the sync cluster

* Release redis 0.24.0 / redis-test 0.3.0 (redis-rs#998)

---------

Co-authored-by: Jacob O'Toole <jacob@jotpot.co.uk>
Co-authored-by: spaceangel <129979014+sp-angel@users.noreply.github.com>
Co-authored-by: eythorhel19 <70693645+eythorhel19@users.noreply.github.com>
Co-authored-by: Eythor Helgason <eytorhelgason@gmail.com>
Co-authored-by: Ruan Petterson <ruan@petterson.eng.br>
Co-authored-by: James Lucas <jaymell@users.noreply.github.com>
Co-authored-by: aumetra <aumetra@cryptolab.net>
Co-authored-by: Josh Leeb-du Toit <mail@joshleeb.com>
Co-authored-by: Ayush <ayushsingh1325@gmail.com>
@enjoy-binbin
Copy link
Contributor

Since v2.6 GA the minimal supported Redis version is 7.2

@LiorKogan
Hi, i see in https://github.com/RedisJSON/RedisJSON/releases/tag/v2.6.0
The minimal Redis version is 6.2, i want to use this in Redis 6.2, do you recommend it?

@LiorKogan
Copy link
Member

I wouldn't recommend using a milestone or a release candidate version in production.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants