RESP3 support #749

nihohit · 2022-12-24T19:27:38Z

Hi, the Value enum supports ATM only RESP2 types. I would like to also add RESP3 support - would you prefer to add that as more enum types to Value (which would be a breaking change), or using some other way?

Additional questions -

Since floats do not implement Eq, we'll need to import ordered-float, or a similar solution. Same for Hash derivation for recursive Maps. Is this acceptable?
In order to implement BigInt, we'll need to import BigInt or a similar solution. Is this acceptable?

remaining work after #757 will be merged:

Streamed aggregated data types
Streamed strings
Mark the map type as being unordered rather than ordered
Allow doubles in scientific E notation
Handle push messages

The text was updated successfully, but these errors were encountered:

jaymell · 2022-12-26T18:25:31Z

I'm not clear enough on the differences to know what's appropriate at this point. But I would proceed as you think appropriate and we can iterate on what you have. Thanks for tackling this!

djc · 2022-12-27T14:21:46Z

The bigint dependency should probably be guarded by a Cargo feature, not quite sure if we should default to having it on or or not. Perhaps we should get rid of Eq for Value instead of pulling in ordered-float?

nihohit · 2022-12-27T15:31:12Z

Would you like RESP3 support to be hidden behind a feature flag? That will prevent build breakages, and ensure that users will opt-in to it, at least in the short term.

djc · 2022-12-27T20:49:26Z

Hmm, not sure what would be appropriate here. Is RESP3 better for existing use cases than RESP2? How do the client and server negotiate which gets used? (Sorry, not that familiar yet with this part of the project.)

nihohit · 2022-12-28T05:42:12Z

The negotiation happens during the HELLO handshake.

Clients that want to handshake using the RESP3 mode need to call the HELLO command and specify the value "3" as the protover argument, like so:

djc · 2023-01-09T13:55:18Z

I think for now RESP3 support should be opt-in on the side of the user, but I don't think it should be a Cargo feature.

nihohit · 2023-01-09T15:06:26Z

This brings us back to the initial question - if you don't use a cargo feature, how do you pass the additional types to the user? Changing Value is a breaking change. Can the users choose whether to receive Value or ValueRESP3?

djc · 2023-01-09T15:09:16Z

Put it in the RedisConnectionInfo somehow?

(I don't necessarily think this change needs to be semver-compatible, anyway.)

nihohit · 2023-01-09T16:29:03Z

That's the question on how the user determines whether to use RESP3 or not.
The question is whether the ConnectionLike traits need to change -
fn req_packed_command(&mut self, cmd: &[u8]) -> RedisResult<Value> - if we modify Value and break exhaustive match on it, that's not a problem. If we don't want to break it, then we need to some other mechanism of returning RESP3 values.

These changes implement all RESP3 types (excluding streamed types). RESP3 can be enabled per connection by adding `?resp3=true` to connection uri. It currently supports PubSub as RESP2 PubSub support in library, but in future PRs it'll support handling normal commands and PubSub messages in one connection. Only `num-bigint` is added as dependency to support `BigNumber` type. Changes made in support of #329 and #749

* Initial implementation of RESP3 (redis-rs#757) These changes implement all RESP3 types (excluding streamed types). RESP3 can be enabled per connection by adding `?resp3=true` to connection uri. It currently supports PubSub as RESP2 PubSub support in library, but in future PRs it'll support handling normal commands and PubSub messages in one connection. Only `num-bigint` is added as dependency to support `BigNumber` type. Changes made in support of redis-rs#329 and redis-rs#749 * Add RESP3 support to cluster connections. (redis-rs#1001) * Resp3 Push Management (redis-rs#898) * squash! * oops! * test invalidation in cluster && introduce client tracking options * introduce basic PubSub functionality to MultiplexedConnection and make tokio sender unbounded channel * fix tests & linter, make PushSender::Tokio as aio feature only * add resp3 to ci branches and fix cluster client tracking option * test dropping and update ci yml * remove unsubscribe fn and introduce unsubscribing by dropping receiver. * fix tests because RedisJson returns responses in an array when it's resp3 * override redisjson cache (it's a temp solution) * add -skip test_module to RESP3 testing and upgrade redis 6.2.4 to 6.2.13 * test json modules with RESP3 and get json fix from main * in redis v7 RedisJson is different with Resp3 🤔 * Implement Pub/Sub in Async Cluster & fmt & remove usage of is_err_and(stable only after v1.70) * don't use sharded pub/sub with redis v6 * use REDIS_VERSION in env instead of using HELLO command to fetch redis version * oops * fix linter * fix fmt and remove benchmark from CI * simplify PushManager and add tokio as non-optional dependency. * get fixes from 220c6a9 * use --test-threads=1 * override redisjson cache (it's a temp solution) * remove get_push_manager from traits & remove push manager from aio::Connection * remove client_tracking_options * remove 0.21.x from rust.yml * add tests for pushmanager * format & move push_info into a variable * change tests according to comments. * apply 6.2.4 changes && fmt * try to fix * remove con_addr & remove pub/sub support in cluster connections * add disconnection handling to sync, mpx, cm && test it * remove push_manager argument from connection creation * better docs * add has_reply function to PushKind * remove no response command support in mpx since it's not used in mpx pub/sub * apply changes from redis-rs#994 * fix tests * Use enum instead of boolean for RESP version. (redis-rs#1012) Since there's a discussion starting about what might become RESP4, this PR will make it easier to add more RESP versions in the future. * Add lock file to keep MSRV constant. (redis-rs#1039) * [opt] preallocate buffer for evalsha in `Script` (redis-rs#1044) * Cluster: fix read from replica & missing slots. (redis-rs#965) * Cluster SlotMap: Allow missing slots. The cluster can still operate when some slots don't have coverage, so there's no reason that the connection won't work. Requests that should be routed to missing slots will be forwarded to a random node. * Fix SlotAddrs not holding all replicas. This change fixes 2 issues - 1. Since SlotAddrs saves at most 1 replica (or none, if `read_from_replica` is false), it means that when there are 2 or more replicas in the cluster, `AllNodes` commands aren't actually sent to all nodes. 2. `SlotAddr::Replica` carries 2 separate semantic meanings - it could mean that the command is a readonly command, or that the user asked to route the command to a replica. The first case should be affected by the `read_from_replica` client flag, but the second case shouldn't - if the user requested that this specific command be routed to the replica, the specific request should override the prior configuration. * Fix multi-slot routes/addresses mismatch. If the `SlotMap` is partial, and some slots don't have addresses, then we need to communicate this back when getting addresses for multi-slot commands. Otherwise we the key indices won't match the addresses, and commands with the wrong keys will be sent to the wrong nodes. * fix typo * fix redis-rs#1045: impl `Clone`/`Copy` for `SetOptions` (redis-rs#1046) * docs: add "connection-manager" cfg attr (redis-rs#1048) * Fix new clippy lints. (redis-rs#1052) Replace Vec reference with a slice, and remove iterator trait when using ExactSizeIterator, since it encompasses it. * Rename Value enum types in order to match Redis RESP names. (redis-rs#779) * Rename Value::Bulk to Value::Array. * Rename Value::Status to Value::SimpleString. * Rename Value::Data to Value::BulkString. * Fix debug names of values. * Fix nightly lints. --------- Co-authored-by: Altan Özlü <5479094+altanozlu@users.noreply.github.com> Co-authored-by: Huxley Hu <framlog@users.noreply.github.com> Co-authored-by: Ahmad <39441506+ahmadbky@users.noreply.github.com> Co-authored-by: ⭐️NINIKA⭐️ <DCNick3@users.noreply.github.com>

* Initial implementation of RESP3 (#757) These changes implement all RESP3 types (excluding streamed types). RESP3 can be enabled per connection by adding `?resp3=true` to connection uri. It currently supports PubSub as RESP2 PubSub support in library, but in future PRs it'll support handling normal commands and PubSub messages in one connection. Only `num-bigint` is added as dependency to support `BigNumber` type. Changes made in support of #329 and #749 * Add RESP3 support to cluster connections. (#1001) * Resp3 Push Management (#898) * squash! * oops! * test invalidation in cluster && introduce client tracking options * introduce basic PubSub functionality to MultiplexedConnection and make tokio sender unbounded channel * fix tests & linter, make PushSender::Tokio as aio feature only * add resp3 to ci branches and fix cluster client tracking option * test dropping and update ci yml * remove unsubscribe fn and introduce unsubscribing by dropping receiver. * fix tests because RedisJson returns responses in an array when it's resp3 * override redisjson cache (it's a temp solution) * add -skip test_module to RESP3 testing and upgrade redis 6.2.4 to 6.2.13 * test json modules with RESP3 and get json fix from main * in redis v7 RedisJson is different with Resp3 🤔 * Implement Pub/Sub in Async Cluster & fmt & remove usage of is_err_and(stable only after v1.70) * don't use sharded pub/sub with redis v6 * use REDIS_VERSION in env instead of using HELLO command to fetch redis version * oops * fix linter * fix fmt and remove benchmark from CI * simplify PushManager and add tokio as non-optional dependency. * get fixes from 220c6a9 * use --test-threads=1 * override redisjson cache (it's a temp solution) * remove get_push_manager from traits & remove push manager from aio::Connection * remove client_tracking_options * remove 0.21.x from rust.yml * add tests for pushmanager * format & move push_info into a variable * change tests according to comments. * apply 6.2.4 changes && fmt * try to fix * remove con_addr & remove pub/sub support in cluster connections * add disconnection handling to sync, mpx, cm && test it * remove push_manager argument from connection creation * better docs * add has_reply function to PushKind * remove no response command support in mpx since it's not used in mpx pub/sub * apply changes from #994 * fix tests * Use enum instead of boolean for RESP version. (#1012) Since there's a discussion starting about what might become RESP4, this PR will make it easier to add more RESP versions in the future. * Rename Value enum types in order to match Redis RESP names. (#779) * Rename Value::Bulk to Value::Array. * Rename Value::Status to Value::SimpleString. * Rename Value::Data to Value::BulkString. * Fix debug names of values. * fix nightly comments. * reintroduce client tracking to tests. * fix merge errors. --------- Co-authored-by: Altan Özlü <5479094+altanozlu@users.noreply.github.com>

* Add lock file to keep MSRV constant. (redis-rs#1039) * [opt] preallocate buffer for evalsha in `Script` (redis-rs#1044) * Cluster: fix read from replica & missing slots. (redis-rs#965) * Cluster SlotMap: Allow missing slots. The cluster can still operate when some slots don't have coverage, so there's no reason that the connection won't work. Requests that should be routed to missing slots will be forwarded to a random node. * Fix SlotAddrs not holding all replicas. This change fixes 2 issues - 1. Since SlotAddrs saves at most 1 replica (or none, if `read_from_replica` is false), it means that when there are 2 or more replicas in the cluster, `AllNodes` commands aren't actually sent to all nodes. 2. `SlotAddr::Replica` carries 2 separate semantic meanings - it could mean that the command is a readonly command, or that the user asked to route the command to a replica. The first case should be affected by the `read_from_replica` client flag, but the second case shouldn't - if the user requested that this specific command be routed to the replica, the specific request should override the prior configuration. * Fix multi-slot routes/addresses mismatch. If the `SlotMap` is partial, and some slots don't have addresses, then we need to communicate this back when getting addresses for multi-slot commands. Otherwise we the key indices won't match the addresses, and commands with the wrong keys will be sent to the wrong nodes. * fix typo * fix redis-rs#1045: impl `Clone`/`Copy` for `SetOptions` (redis-rs#1046) * docs: add "connection-manager" cfg attr (redis-rs#1048) * Fix new clippy lints. (redis-rs#1052) Replace Vec reference with a slice, and remove iterator trait when using ExactSizeIterator, since it encompasses it. * Async cluster connection: Improve handling of missing connections (redis-rs#968) * aio::ClusterConnection: Report missing connections. This change should ensure that if a connection wasn't found, after redirecting to node, or when no random connection is available, then the cluster connection will refresh slots. * Add sleep to refresh slots action. * Cancel redirects after disconnects. If a redirected request reaches a disconnected node, the redirection will be cancelled, and the routing will revert to the original routing. * Move OperationTarget to Err side of result, and reduce generics. OperationTarget is used only on errors, so it should be in the `Err` case only. The added generics were used to hide a single type. * Handle disconnect from all nodes. If the async cluster connection completely disconnects from all nodes in the server, it will try again to connect to the inital nodes that were provided upon creation. This prevents a situation where the client removes connections incrementally, until the connection map is completely empty, and there are no connections to refresh slots on. * Appease Clippy (redis-rs#1061) * Reconnect on parsing errors. Parsing errors mean that the connection received a response it doesn't know how to handle. This means that it cannot make sense of the next values sent over the connection, and the connection must be replaced. related: redis-rs#984 (comment) * Save reconnected connections during retries. This change ensures that reconnect attempts that happen during retries, or new connections that happen after MOVED/ASKING errors, will be saved instead of constantly reconnecting until slots are refreshed. * [fix] make `Pipeline` handle returned bulks correctly * Update mio dependency due to vulnerability. (redis-rs#1064) GHSA-r8w9-5wcg-vfj7 * Simplify Sink polling logic. This removes unnecessary `match`es and `map` from the code, and moves the usage of `poll_recover` into `poll_flush`, so as not to block new requests while trying to recover a connection. * Remove the usage of aio::Connection in tests. `aio::Connection` is deprecated, we should test `aio::MultiplexedConnection` instead. * Fail CI if lock file isn't updated. * Handle server errors in array response. Currently server errors stop the parser and return a RedisError. This caused errors that are returned inside an array, such as transaction errors, to cause the rest of the array to not be parsed. This is solved by adding an internal value type that includes the server errors, so when parsing to the internal value type, the array will finish parsing, and then extracting the error. * Create a server error type. * Separate parsing errors from general response errors. Parsing errors are client-side errors, that are caused by bad output from the server. Response errors are server-side errors, caused by bad input from the user / client. Parse errors cause the client to be in an unrecoverable state. response errors are OK. * Release redis 0.25.0 / redis-test 0.4.0 * Update test version (redis-rs#1071) * Fix ambiguity in examples * Upgrade to socket2 0.5 * Avoid library dependency on futures-time * Merge the `resp3` branch. (redis-rs#1058) * Initial implementation of RESP3 (redis-rs#757) These changes implement all RESP3 types (excluding streamed types). RESP3 can be enabled per connection by adding `?resp3=true` to connection uri. It currently supports PubSub as RESP2 PubSub support in library, but in future PRs it'll support handling normal commands and PubSub messages in one connection. Only `num-bigint` is added as dependency to support `BigNumber` type. Changes made in support of redis-rs#329 and redis-rs#749 * Add RESP3 support to cluster connections. (redis-rs#1001) * Resp3 Push Management (redis-rs#898) * squash! * oops! * test invalidation in cluster && introduce client tracking options * introduce basic PubSub functionality to MultiplexedConnection and make tokio sender unbounded channel * fix tests & linter, make PushSender::Tokio as aio feature only * add resp3 to ci branches and fix cluster client tracking option * test dropping and update ci yml * remove unsubscribe fn and introduce unsubscribing by dropping receiver. * fix tests because RedisJson returns responses in an array when it's resp3 * override redisjson cache (it's a temp solution) * add -skip test_module to RESP3 testing and upgrade redis 6.2.4 to 6.2.13 * test json modules with RESP3 and get json fix from main * in redis v7 RedisJson is different with Resp3 🤔 * Implement Pub/Sub in Async Cluster & fmt & remove usage of is_err_and(stable only after v1.70) * don't use sharded pub/sub with redis v6 * use REDIS_VERSION in env instead of using HELLO command to fetch redis version * oops * fix linter * fix fmt and remove benchmark from CI * simplify PushManager and add tokio as non-optional dependency. * get fixes from 220c6a9 * use --test-threads=1 * override redisjson cache (it's a temp solution) * remove get_push_manager from traits & remove push manager from aio::Connection * remove client_tracking_options * remove 0.21.x from rust.yml * add tests for pushmanager * format & move push_info into a variable * change tests according to comments. * apply 6.2.4 changes && fmt * try to fix * remove con_addr & remove pub/sub support in cluster connections * add disconnection handling to sync, mpx, cm && test it * remove push_manager argument from connection creation * better docs * add has_reply function to PushKind * remove no response command support in mpx since it's not used in mpx pub/sub * apply changes from redis-rs#994 * fix tests * Use enum instead of boolean for RESP version. (redis-rs#1012) Since there's a discussion starting about what might become RESP4, this PR will make it easier to add more RESP versions in the future. * Rename Value enum types in order to match Redis RESP names. (redis-rs#779) * Rename Value::Bulk to Value::Array. * Rename Value::Status to Value::SimpleString. * Rename Value::Data to Value::BulkString. * Fix debug names of values. * fix nightly comments. * reintroduce client tracking to tests. * fix merge errors. --------- Co-authored-by: Altan Özlü <5479094+altanozlu@users.noreply.github.com> * fix flakey test * Fix linter. * fix cargo lock. --------- Co-authored-by: Huxley Hu <framlog@users.noreply.github.com> Co-authored-by: Ahmad <39441506+ahmadbky@users.noreply.github.com> Co-authored-by: ⭐️NINIKA⭐️ <DCNick3@users.noreply.github.com> Co-authored-by: James Lucas <jaymell@users.noreply.github.com> Co-authored-by: framlog <framlog@gmail.com> Co-authored-by: Neo Sun <huachuang20@gmail.com> Co-authored-by: Dirkjan Ochtman <dirkjan@ochtman.nl> Co-authored-by: Altan Özlü <5479094+altanozlu@users.noreply.github.com>

shachlanAmazon mentioned this issue Mar 1, 2023

Initial implementation of RESP3 #757

Merged

nihohit mentioned this issue Jul 25, 2023

Resp3 Push Management #898

Merged

5 tasks

nihohit mentioned this issue Mar 28, 2024

Support RESP3 protocol #329

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RESP3 support #749

RESP3 support #749

nihohit commented Dec 24, 2022 •

edited

jaymell commented Dec 26, 2022

djc commented Dec 27, 2022

nihohit commented Dec 27, 2022

djc commented Dec 27, 2022

nihohit commented Dec 28, 2022

djc commented Jan 9, 2023

nihohit commented Jan 9, 2023

djc commented Jan 9, 2023 •

edited

nihohit commented Jan 9, 2023

RESP3 support #749

RESP3 support #749

Comments

nihohit commented Dec 24, 2022 • edited

jaymell commented Dec 26, 2022

djc commented Dec 27, 2022

nihohit commented Dec 27, 2022

djc commented Dec 27, 2022

nihohit commented Dec 28, 2022

djc commented Jan 9, 2023

nihohit commented Jan 9, 2023

djc commented Jan 9, 2023 • edited

nihohit commented Jan 9, 2023

nihohit commented Dec 24, 2022 •

edited

djc commented Jan 9, 2023 •

edited