Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CCR] Refactore auto follow coordinator #35895

Closed

Conversation

Projects
None yet
4 participants
@martijnvg
Copy link
Member

commented Nov 26, 2018

  • AutofollowCoordinator now fetches cluster states per remote cluster.
  • AutofollowCoordinator makes use of the wait_for_metadata_version
    feature in cluster state API.
  • Removed the poll_interval setting, because it is no longer needed.
    Waiting is done via cluster state api for changes in remote clusters and
    auto follow pattern changes in the local cluster are immediately processed.
  • Improved auto follow stats to keep track of last time the remote clusters
    were checked for changes per remote cluster.

Relates to #33007

[CCR] Refactored auto follow coordinator
* AutofollowCoordinator now fetches cluster states per remote cluster.
* AutofollowCoordinator makes use of the `wait_for_metadata_version`
  feature in cluster state API.
* Removed the poll_interval setting, because it is no longer needed.
  Waiting is done via cluster state api for changes in remote clusters and
  auto follow pattern changes in the local cluster are immediately processed.
* Improved auto follow stats to keep track of last time the remote clusters
  were checked for changes per remote cluster.

Relates to #33007
@elasticmachine

This comment has been minimized.

Copy link

commented Nov 26, 2018

@@ -33,6 +33,9 @@
private static final ParseField RECENT_AUTO_FOLLOW_ERRORS = new ParseField("recent_auto_follow_errors");
private static final ParseField LEADER_INDEX = new ParseField("leader_index");
private static final ParseField AUTO_FOLLOW_EXCEPTION = new ParseField("auto_follow_exception");
private static final ParseField TRACKING_REMOTE_CLUSTERS = new ParseField("tracking_remote_clusters");
private static final ParseField CLUSTER_NAME = new ParseField("cluster_name");
private static final ParseField TIME_SINCE_LAST_AUTO_FOLLOW_STARTED_MILLIS = new ParseField("time_since_last_auto_follow_started_millis");

This comment has been minimized.

Copy link
@martijnvg

martijnvg Nov 26, 2018

Author Member

Maybe time_since_last_remote_cluster_state_api_call is a better name here.

martijnvg added some commits Nov 26, 2018

@martijnvg

This comment has been minimized.

Copy link
Member Author

commented Nov 27, 2018

run the gradle build tests 2

@jasontedor
Copy link
Member

left a comment

Heya @martijnvg I think this PR can be broken down into smaller PRs. I would suggest the following changes:

  • remove the poll_interval setting, hard coded the poll_interval to the current default (this will show up a separate change in the change log, that's a good thing)
  • add the use of the wait_for_metdata_version parameter on the cluster state API, replacing the hard coded poll_interval
  • add the last requested time to the stats API

martijnvg added a commit to martijnvg/elasticsearch that referenced this pull request Nov 29, 2018

[CCR] Refactor AutoFollowCoordinator to track leader indices per remo…
…te cluster

and replaced poll interval setting with a hardcoded poll interval.
The hard coded interval will be removed in a follow up change to make
use of cluster state API's wait_for_metatdata_version.

Originates from elastic#35895
Relates to elastic#33007
@martijnvg

This comment has been minimized.

Copy link
Member Author

commented Nov 29, 2018

Closing this PR. The changes in this PR will be added in several other PRs.

@martijnvg martijnvg closed this Nov 29, 2018

martijnvg added a commit to martijnvg/elasticsearch that referenced this pull request Nov 29, 2018

[CCR] Refactor AutoFollowCoordinator to track leader indices per remo…
…te cluster

and replaced poll interval setting with a hardcoded poll interval.
The hard coded interval will be removed in a follow up change to make
use of cluster state API's wait_for_metatdata_version.

Originates from elastic#35895
Relates to elastic#33007

martijnvg added a commit that referenced this pull request Dec 5, 2018

Refactor AutoFollowCoordinator to track leader indices per remote clu…
…ster (#36031)

and replaced poll interval setting with a hardcoded poll interval.
The hard coded interval will be removed in a follow up change to make
use of cluster state API's wait_for_metatdata_version.

Before the auto following was bootstrapped from thread pool scheduler,
but now auto followers for new remote clusters are bootstrapped when
a new cluster state is published.

Originates from #35895
Relates to #33007

martijnvg added a commit that referenced this pull request Dec 5, 2018

Refactor AutoFollowCoordinator to track leader indices per remote clu…
…ster (#36031)

and replaced poll interval setting with a hardcoded poll interval.
The hard coded interval will be removed in a follow up change to make
use of cluster state API's wait_for_metatdata_version.

Before the auto following was bootstrapped from thread pool scheduler,
but now auto followers for new remote clusters are bootstrapped when
a new cluster state is published.

Originates from #35895
Relates to #33007

martijnvg added a commit to martijnvg/elasticsearch that referenced this pull request Dec 5, 2018

[CCR] Change AutofollowCoordinator to use wait_for_metadata_version
Changed AutofollowCoordinator makes use of the wait_for_metadata_version
feature in cluster state API and removed hard coded poll interval.

Originates from elastic#35895
Relates to elastic#33007

atorok added a commit to cdschneider/elasticsearch that referenced this pull request Dec 6, 2018

Refactor AutoFollowCoordinator to track leader indices per remote clu…
…ster (elastic#36031)

and replaced poll interval setting with a hardcoded poll interval.
The hard coded interval will be removed in a follow up change to make
use of cluster state API's wait_for_metatdata_version.

Before the auto following was bootstrapped from thread pool scheduler,
but now auto followers for new remote clusters are bootstrapped when
a new cluster state is published.

Originates from elastic#35895
Relates to elastic#33007

martijnvg added a commit that referenced this pull request Dec 12, 2018

[CCR] Change AutofollowCoordinator to use wait_for_metadata_version (#…
…36264)

Changed AutofollowCoordinator makes use of the wait_for_metadata_version
feature in cluster state API and removed hard coded poll interval.

Originates from #35895
Relates to #33007

martijnvg added a commit that referenced this pull request Dec 12, 2018

[CCR] Change AutofollowCoordinator to use wait_for_metadata_version (#…
…36264)

Changed AutofollowCoordinator makes use of the wait_for_metadata_version
feature in cluster state API and removed hard coded poll interval.

Originates from #35895
Relates to #33007

martijnvg added a commit to martijnvg/elasticsearch that referenced this pull request Dec 12, 2018

[CCR] Add time since last auto follow fetch to auto follow stats
For each remote cluster the auto follow coordinator, starts an auto
follower that checks the remote cluster state and determines whether an
index needs to be auto followed. The time since last auto follow is
reported per remote cluster and gives insight whether the auto follow
process is alive.

Relates to elastic#33007
Originates from elastic#35895

martijnvg added a commit that referenced this pull request Dec 17, 2018

[CCR] Add time since last auto follow fetch to auto follow stats (#36542
)

For each remote cluster the auto follow coordinator, starts an auto
follower that checks the remote cluster state and determines whether an
index needs to be auto followed. The time since last auto follow is
reported per remote cluster and gives insight whether the auto follow
process is alive.

Relates to #33007
Originates from #35895

martijnvg added a commit that referenced this pull request Dec 17, 2018

[CCR] Add time since last auto follow fetch to auto follow stats (#36542
)

For each remote cluster the auto follow coordinator, starts an auto
follower that checks the remote cluster state and determines whether an
index needs to be auto followed. The time since last auto follow is
reported per remote cluster and gives insight whether the auto follow
process is alive.

Relates to #33007
Originates from #35895

@colings86 colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.