Skip to content

Kubernetes service discovery breaking change in version 29.0.0#17529

Closed
m17kea wants to merge 1 commit intoapache:masterfrom
m17kea:patch-1
Closed

Kubernetes service discovery breaking change in version 29.0.0#17529
m17kea wants to merge 1 commit intoapache:masterfrom
m17kea:patch-1

Conversation

@m17kea
Copy link

@m17kea m17kea commented Dec 2, 2024

Fixes #15942

Description

This PR updates the Apache Druid documentation to highlight a breaking change in Kubernetes service discovery behavior introduced in version 29. Specifically, the format of the timestamps used in the <cluster-identifier>-leaderelection-coordinator and <cluster-identifier>-leaderelection-overlord ConfigMaps was updated to a new format that was not documented. This change caused errors during leader election and service discovery due to mismatched date parsing expectations.

The goal of this PR is to ensure that users upgrading to version 29 are aware of this breaking change and can make the necessary adjustments to avoid issues in their Druid clusters.

Documentation Changes
Added a note in the Upgrade notes documentation for version 29 about the updated date format in ConfigMaps.
Provided an example of the command required to help users update their clusters accordingly.

Release Note

This PR updates the documentation to warn users about a breaking change in Kubernetes service discovery introduced in version 29. The date format in leader election ConfigMaps (<cluster-identifier>-leaderelection-coordinator and <cluster-identifier>-leaderelection-overlord) was updated to a stricter ISO-8601-like format. Users should ensure they delete the old config mapss once upgrade to version 29.0.0 and above so they are recreated with the correct date format.


Key changed/added classes in this PR Documentation updates only, no code changes.

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@m17kea
Copy link
Author

m17kea commented Dec 2, 2024

Likely to have been introduced by #15449

@georgew5656
Copy link
Contributor

georgew5656 commented Jan 7, 2025

Likely to have been introduced by #15449

this looks good to me, @m17kea can you add a note that the operator might need to delete the configmap multiple times? for example if they have two coordinators or overlords running and are doing a rolling update, and one of the old coordinators/overlords regains leadership after the configmap is deleted

@georgew5656 georgew5656 self-requested a review January 7, 2025 20:50

#### Changes to date format in kubernetes service discovery

Druid 29.0.0 includes a breaking change for those using kubernetes service discovery. The date format used in the config maps used for leader election of coordinators and overlords has changed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please describe the error a little more or maybe include an example of the encountered error message.

@github-actions
Copy link

github-actions bot commented May 2, 2025

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

@github-actions github-actions bot added the stale label May 2, 2025
@github-actions
Copy link

This pull request/issue has been closed due to lack of activity. If you think that
is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions bot closed this May 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Coordinator crashes after upgrading to 29.0.0

3 participants