Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] Make sure redis sync context and async context connect to the same redis instance #42040

Merged
merged 2 commits into from
Dec 20, 2023

Conversation

jjyao
Copy link
Contributor

@jjyao jjyao commented Dec 20, 2023

Why are these changes needed?

We are using ip for redis sync context connection but domain name for redis async context so there is a chance redis async context doesn't connect to the master.

I think what can happen is that during RedisContext::Connect(), the domain name is resolved to two ips (ip[0] is master, ip[1] is replica). sync context is connected to ip[0] so it's fine while async context is connected using domain name so it might be connected to ip[1] which is a replica. Later on we have check to make sure we are connected to the master but we only do the check for sync context so we don't realize that the async context is connected to the replica. Then when we use async context to send HGET request, replica will return MOVED (https://redis.io/commands/readonly/) to redirect us to the master and since we don't handle MOVED at that time, it failed.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

…dis instance

Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com>
@jjyao jjyao requested a review from a team as a code owner December 20, 2023 17:37
@vitsai
Copy link
Contributor

vitsai commented Dec 20, 2023

oh nice

Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com>
@jjyao jjyao merged commit b908d57 into ray-project:master Dec 20, 2023
10 checks passed
@jjyao jjyao deleted the jjyao/redis branch December 20, 2023 20:34
jjyao added a commit that referenced this pull request Dec 21, 2023
…same redis instance (#42040)

We are using ip for redis sync context connection but domain name for redis async context so there is a chance redis async context doesn't connect to the master.

Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com>
architkulkarni pushed a commit that referenced this pull request Dec 21, 2023
…same redis instance (#42040) (#42073)

We are using ip for redis sync context connection but domain name for redis async context so there is a chance redis async context doesn't connect to the master.



---------

Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com>
vickytsang pushed a commit to ROCm/ray that referenced this pull request Jan 12, 2024
…same redis instance (ray-project#42040)

We are using ip for redis sync context connection but domain name for redis async context so there is a chance redis async context doesn't connect to the master.

Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants