Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Discovery GCE Pagination Lack #83783

Closed
cdmbr opened this issue Feb 10, 2022 · 1 comment · Fixed by #83785
Closed

Discovery GCE Pagination Lack #83783

cdmbr opened this issue Feb 10, 2022 · 1 comment · Fixed by #83785
Labels
>bug :Distributed/Discovery-Plugins Anything related to our integration plugins with EC2, GCP and Azure Team:Distributed Meta label for distributed team

Comments

@cdmbr
Copy link
Contributor

cdmbr commented Feb 10, 2022 •
edited

Elasticsearch Version

All

Installed Plugins

discovery-gce

Java Version

bundled

OS Version

any

Problem Description

Currently, discovery-gce plugin tries to obtain all instances for a given project/zone in one shot.
Recently I've faced in my company (that had ~ 700 instances running at the time) a weird behavior of this plugin. The google api that this plugin uses paginates results in chunks of 500 items. The current implementation is ignoring if there's a next page available.

Steps to Reproduce

  • run the node discovery process in zones that have more than 500 instances

Logs (if relevant)

No response
@cdmbr cdmbr added >bug needs:triage Requires assignment of a team area label labels Feb 10, 2022
@cdmbr cdmbr mentioned this issue Feb 10, 2022
Merged
@DaveCTurner DaveCTurner added the :Distributed/Discovery-Plugins Anything related to our integration plugins with EC2, GCP and Azure label Feb 10, 2022
@elasticmachine elasticmachine added the Team:Distributed Meta label for distributed team label Feb 10, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@DaveCTurner DaveCTurner added Team:Distributed Meta label for distributed team and removed Team:Distributed Meta label for distributed team needs:triage Requires assignment of a team area label labels Feb 10, 2022
arteam pushed a commit that referenced this issue Feb 15, 2022
@cdmbr
[GCE Discovery] Correcly handle large zones with 500 or more instances (
2bcc03d 
#83785)

Discovery gce plugin has some issues while running in large pools of vms instances.

This pr attempts to solve it.

Closes #83783
arteam pushed a commit to arteam/elasticsearch that referenced this issue Feb 15, 2022
@cdmbr @arteam
[GCE Discovery] Correcly handle large zones with 500 or more instances (
b9e126d 
elastic#83785)

Discovery gce plugin has some issues while running in large pools of vms instances.

This pr attempts to solve it.

Closes elastic#83783
arteam pushed a commit to arteam/elasticsearch that referenced this issue Feb 15, 2022
@cdmbr @arteam
[GCE Discovery] Correcly handle large zones with 500 or more instances (
136d797 
elastic#83785)

Discovery gce plugin has some issues while running in large pools of vms instances.

This pr attempts to solve it.

Closes elastic#83783
arteam added a commit that referenced this issue Feb 15, 2022
@arteam @cdmbr
[GCE Discovery] Correcly handle large zones with 500 or more instances (
47839fb 
#83785) (#83978)

Discovery gce plugin has some issues while running in large pools of vms instances.

This pr attempts to solve it.

Closes #83783

Co-authored-by: Claudio Marins <cdmbr@users.noreply.github.com>
arteam added a commit that referenced this issue Feb 15, 2022
@arteam @cdmbr
[GCE Discovery] Correcly handle large zones with 500 or more instances (
e4cfe2c 
#83785) (#83979)

Discovery gce plugin has some issues while running in large pools of vms instances.

This pr attempts to solve it.

Closes #83783

Co-authored-by: Claudio Marins <cdmbr@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>bug :Distributed/Discovery-Plugins Anything related to our integration plugins with EC2, GCP and Azure Team:Distributed Meta label for distributed team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[GCE Discovery] Correcly handle large zones with 500 or more instances cdmbr/elasticsearch
3 participants
@DaveCTurner @cdmbr @elasticmachine