Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: no switchover candidate found with member state "streaming" (#1992) #2515

Merged
merged 3 commits into from
Jan 24, 2024

Conversation

bootc
Copy link
Contributor

@bootc bootc commented Jan 14, 2024

Fixes #1992

@hughcapet
Copy link
Member

There is another possible state of a running replica - "in archive recovery"
patroni/patroni@d46ca88

@bootc bootc force-pushed the fix/switchover_member_state_streaming branch from bfe2830 to efc3b1f Compare January 22, 2024 21:16
@FxKu
Copy link
Member

FxKu commented Jan 23, 2024

👍

continue
}

if slices.Contains([]string{"running", "streaming", "in archive recovery"}, member.State) {
Copy link
Member

@macedigital macedigital Jan 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The next part of the code sorts candidates by smallest amount of lag. My only concern with merging this change is that it might cause a switchover to a replica lagging behind significantly, as lag is not guaranteed to be 0.

Overall, this is still better than what we have now, though.

if len(candidates) > 0 {
sort.Slice(candidates, func(i, j int) bool {
return candidates[i].Lag < candidates[j].Lag
})
return spec.NamespacedName{Namespace: master.Namespace, Name: candidates[0].Name}, nil
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Patroni will prevent it if this is the case.

Copy link
Member

@macedigital macedigital left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's merge this PR and make a new release for fixing the underlying issue.

@@ -75,6 +75,13 @@ func TestGetSwitchoverCandidate(t *testing.T) {
expectedCandidate: spec.NamespacedName{},
expectedError: fmt.Errorf("no switchover candidate found"),
},
{
subtest: "replicas with streaming state",
clusterJson: `{"members": [{"name": "acid-test-cluster-0", "role": "leader", "state": "running", "api_url": "http://192.168.100.1:8008/patroni", "host": "192.168.100.1", "port": 5432, "timeline": 1}, {"name": "acid-test-cluster-1", "role": "replica", "state": "streaming", "api_url": "http://192.168.100.2:8008/patroni", "host": "192.168.100.2", "port": 5432, "timeline": 1, "lag": 5}, {"name": "acid-test-cluster-2", "role": "replica", "state": "streaming", "api_url": "http://192.168.100.3:8008/patroni", "host": "192.168.100.3", "port": 5432, "timeline": 1, "lag": 2}]}`,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might want to add test cases to capture "in archive recovery" mode, eventually.

@hughcapet
Copy link
Member

👍

@FxKu FxKu merged commit 8f31399 into zalando:master Jan 24, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

No switchover candidate found
4 participants