Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Propagate readiness status to all datacenters #1826

Merged
merged 1 commit into from
Mar 4, 2024

Conversation

piotrrzysko
Copy link
Member

@piotrrzysko piotrrzysko commented Feb 29, 2024

Intro

This is part of the feature of falling back to remote datacenters when the local one is unhealthy during publishing. Considering that hermes-frontend has access only to ZooKeeper in its local datacenter, it needs to have information there on the readiness status of all datacenters to select only those that are ready. Before this change, only the readiness status of the local datacenter was stored in the local ZooKeeper.

Migration

  • To ensure backward compatibility with previous versions, this PR introduces a new ZooKeeper path (/datacenter-readiness) to store readiness status, instead of using the old path (/frontend/readiness). Thus, rollbacks to older versions are possible without any additional changes.
  • The new path (/datacenter-readiness) is populated on the first and subsequent readiness status changes. Versions including this change do not respect values set in the previous path (/frontend/readiness). If the new path does not exist or is empty, Hermes assumes that all datacenters are ready.
  • The old path is not automatically removed. Therefore, we recommend manually deleting it after migration.

This is part of the feature of falling back to remote datacenters when the local one is unhealthy during publishing. Considering that hermes-frontend has access only to ZooKeeper in its local datacenter, it needs to have information there on the readiness status of all datacenters to select only those that are ready. Before this change, only the readiness status of the local datacenter was stored in the local ZooKeeper.
Copy link
Collaborator

@moscicky moscicky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very smooth change. I would only suggest to mention the migration process (or lack of it) and whether the change is safe to rollback in the PR description. As far as I understand defaults would always return ready so we can just mention that that:

  1. true values will be persisted in ZK on first save
  2. old readiness path should be removed manually because it will become orphaned

@@ -52,7 +52,6 @@ public int hashCode() {

public enum ReadinessStatus {
READY,
NOT_READY,
UNDEFINED
NOT_READY
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏻

@piotrrzysko piotrrzysko marked this pull request as ready for review March 4, 2024 08:15
@piotrrzysko
Copy link
Member Author

I would only suggest to mention the migration process (or lack of it) and whether the change is safe to rollback in the PR description.

Good point. Added.

@piotrrzysko piotrrzysko merged commit f2cef4c into master Mar 4, 2024
28 checks passed
@piotrrzysko piotrrzysko deleted the readiness-replication-to-all-datacenters branch March 4, 2024 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants