fix multi_cluster_management check timeout #7377
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hslightdb
force-pushed
the
fix_timeout
branch
from
a1c72cb
to
89ae723
Compare
|
The change seems reasonable in general, but could you explain a bit more how it fixes the issue you were seeing? |
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #7377 +/- ##
==========================================
- Coverage 89.51% 80.17% -9.34%
==========================================
Files 280 280
Lines 60304 60299 -5
Branches 7505 7505
==========================================
- Hits 53979 48343 -5636
- Misses 4160 9204 +5044
- Partials 2165 2752 +587 |
when have two localhost in /etc/hosts, socket will changed after MultiConnectionStatePoll( from first localhost to next locahost), but it will not reset wait events, then it will wait forever. |
|
Thank you for this fix. afaict it's actually not just a bug during testing then, it could also occur in normal usage when a hostname in pg_dist_node resolves to multiple IPs. |
JelteF
approved these changes
Jan 10, 2024
JelteF
pushed a commit
that referenced
this pull request
Apr 16, 2024
…7377) When there are multiple localhost entries in /etc/hosts like following /etc/hosts: ``` 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 localhost ``` multi_cluster_management check will failed: ``` @@ -857,20 +857,21 @@ ERROR: group 14 already has a primary node -- check that you can add secondaries and unavailable nodes to a group SELECT groupid AS worker_2_group FROM pg_dist_node WHERE nodeport = :worker_2_port \gset SELECT 1 FROM master_add_node('localhost', 9998, groupid => :worker_1_group, noderole => 'secondary'); ?column? ---------- 1 (1 row) SELECT 1 FROM master_add_node('localhost', 9997, groupid => :worker_1_group, noderole => 'unavailable'); +WARNING: could not establish connection after 5000 ms ?column? ---------- 1 (1 row) ``` This actually isn't just a problem in test environments, but could occur as well during actual usage when a hostname in pg_dist_node resolves to multiple IPs and one of those IPs is unreachable. Postgres will then automatically continue with the next IP, but Citus should listen for events on the new socket. Not on the old one. Co-authored-by: chuhx43211 <chuhx43211@hundsun.com> (cherry picked from commit 9a91136)
JelteF
pushed a commit
that referenced
this pull request
Apr 16, 2024
…7377) When there are multiple localhost entries in /etc/hosts like following /etc/hosts: ``` 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 localhost ``` multi_cluster_management check will failed: ``` @@ -857,20 +857,21 @@ ERROR: group 14 already has a primary node -- check that you can add secondaries and unavailable nodes to a group SELECT groupid AS worker_2_group FROM pg_dist_node WHERE nodeport = :worker_2_port \gset SELECT 1 FROM master_add_node('localhost', 9998, groupid => :worker_1_group, noderole => 'secondary'); ?column? ---------- 1 (1 row) SELECT 1 FROM master_add_node('localhost', 9997, groupid => :worker_1_group, noderole => 'unavailable'); +WARNING: could not establish connection after 5000 ms ?column? ---------- 1 (1 row) ``` This actually isn't just a problem in test environments, but could occur as well during actual usage when a hostname in pg_dist_node resolves to multiple IPs and one of those IPs is unreachable. Postgres will then automatically continue with the next IP, but Citus should listen for events on the new socket. Not on the old one. Co-authored-by: chuhx43211 <chuhx43211@hundsun.com> (cherry picked from commit 9a91136)
JelteF
pushed a commit
that referenced
this pull request
Apr 16, 2024
…7377) When there are multiple localhost entries in /etc/hosts like following /etc/hosts: ``` 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 localhost ``` multi_cluster_management check will failed: ``` @@ -857,20 +857,21 @@ ERROR: group 14 already has a primary node -- check that you can add secondaries and unavailable nodes to a group SELECT groupid AS worker_2_group FROM pg_dist_node WHERE nodeport = :worker_2_port \gset SELECT 1 FROM master_add_node('localhost', 9998, groupid => :worker_1_group, noderole => 'secondary'); ?column? ---------- 1 (1 row) SELECT 1 FROM master_add_node('localhost', 9997, groupid => :worker_1_group, noderole => 'unavailable'); +WARNING: could not establish connection after 5000 ms ?column? ---------- 1 (1 row) ``` This actually isn't just a problem in test environments, but could occur as well during actual usage when a hostname in pg_dist_node resolves to multiple IPs and one of those IPs is unreachable. Postgres will then automatically continue with the next IP, but Citus should listen for events on the new socket. Not on the old one. Co-authored-by: chuhx43211 <chuhx43211@hundsun.com> (cherry picked from commit 9a91136)
JelteF
pushed a commit
that referenced
this pull request
Apr 17, 2024
…7377) When there are multiple localhost entries in /etc/hosts like following /etc/hosts: ``` 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 localhost ``` multi_cluster_management check will failed: ``` @@ -857,20 +857,21 @@ ERROR: group 14 already has a primary node -- check that you can add secondaries and unavailable nodes to a group SELECT groupid AS worker_2_group FROM pg_dist_node WHERE nodeport = :worker_2_port \gset SELECT 1 FROM master_add_node('localhost', 9998, groupid => :worker_1_group, noderole => 'secondary'); ?column? ---------- 1 (1 row) SELECT 1 FROM master_add_node('localhost', 9997, groupid => :worker_1_group, noderole => 'unavailable'); +WARNING: could not establish connection after 5000 ms ?column? ---------- 1 (1 row) ``` This actually isn't just a problem in test environments, but could occur as well during actual usage when a hostname in pg_dist_node resolves to multiple IPs and one of those IPs is unreachable. Postgres will then automatically continue with the next IP, but Citus should listen for events on the new socket. Not on the old one. Co-authored-by: chuhx43211 <chuhx43211@hundsun.com> (cherry picked from commit 9a91136)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DESCRIPTION: Fixes bug when hostname in pg_dist_node resolves to multiple IPs
when repeat localhost in /etc/hosts like following
/etc/hosts:
multi_cluster_management check will failed:
this pr is attempt to fix it