Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve management of Elasticsearch client socket pools #137734

Closed
gsoldevila opened this issue Aug 1, 2022 · 2 comments · Fixed by #137748
Closed

Improve management of Elasticsearch client socket pools #137734

gsoldevila opened this issue Aug 1, 2022 · 2 comments · Fixed by #137748
Labels
Feature:elasticsearch performance Supportability Improve our (devs, SREs, support eng, users) ability to troubleshoot/self-service product better. Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@gsoldevila
Copy link
Contributor

gsoldevila commented Aug 1, 2022

Currently, every time we instantiate elasticsearch-js's Client class, we are creating a new HTTP(s) Agent that is capable of connecting to multiple origins, and managing its own pool of sockets.

In practice, what that means is if we have two Client instances that connect to the same ES nodes, they will NOT be sharing the same pool of sockets. We will not benefit from idle sockets, and eventually we will have to establish a few more secure connections to ES (aka more TCP + TLS handshakes). Also, it makes things very difficult if we want to track / limit maximum number of open connections, as we don't have control of which Agent instances / sockets pools exist.

We want to be able to group the Client instances logically by domain (e.g. those that are intended for data retrieval, those that are for monitoring purposes, etc). This will allow:

  • controlling which pools of sockets exist at a certain point in time.
  • reusing HttpConnections (less pools means less active sockets, less idle sockets, and less TCP+TLS handshakes).
@gsoldevila gsoldevila added Feature:elasticsearch Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc performance Supportability Improve our (devs, SREs, support eng, users) ability to troubleshoot/self-service product better. labels Aug 1, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@rudolf
Copy link
Contributor

rudolf commented Aug 4, 2022

Although we can work around it for now, I've created an upstream issue for Elasticsearch-js elastic/elasticsearch-js#1740

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:elasticsearch performance Supportability Improve our (devs, SREs, support eng, users) ability to troubleshoot/self-service product better. Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants