Add networking guide docs #138119

mhl-b · 2025-11-14T18:22:50Z

A high level overview of networking for the distributed architecture guide. ES-7886

elasticsearchmachine · 2025-11-14T18:24:00Z

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

DaveCTurner

I left a few factual nits, nothing major. It needs some editorial polishing too - can we use any of our fancy new AI tools to help with that?

DaveCTurner · 2025-11-15T07:48:08Z

docs/internal/DistributedArchitectureGuide.md

+be HTTP spec compliant, Elastic is not a webserver. We support GET requests with
+payload (some old proxies might drop content), requests cannot be cached by


Maybe mention that in these cases we also support the same API with a different verb, normally POST, for clients who cannot send a GET-with-body.

DaveCTurner · 2025-11-15T07:58:20Z

docs/internal/DistributedArchitectureGuide.md

+
+HTTP transport provides two options for content processing: aggregate fully and
+incremental. Aggregated content is a preferable choice for a small messages that
+cannot be parsed incrementally (like JSON). But aggregation has drawbacks, it


cannot be parsed incrementally (like JSON)

Mmm sorta depends what you mean by "incrementally" but I'd say that the SAX-style parsing we do is in fact working incrementally. It doesn't really make sense to start parsing before we've received the whole body, but that's different from saying "cannot".

DaveCTurner · 2025-11-15T08:01:59Z

docs/internal/DistributedArchitectureGuide.md

+The job of the REST handler is to parse and validate HTTP request and construct
+typed version of request, often Transport request (see Transport section below).


Might be worth pointing out here that (if security is enabled then) authentication happens before the REST handler but authorization happens after (when entering the transport layer).

DaveCTurner · 2025-11-15T08:02:41Z

docs/internal/DistributedArchitectureGuide.md

+
+`Transport` is an umbrella term for a node-to-node communication. It's a
+TCP-based custom binary protocol. Every node in a cluster is a client and server
+at the same time. Node-to-node communication never uses HTTP transport.


never

except for reindex-from-remote :)

DaveCTurner · 2025-11-15T08:04:35Z

docs/internal/DistributedArchitectureGuide.md

+connections for different purposes: ping, node-state, bulks, etc. Pool structure
+is defined in `ConnectionProfile` class.
+
+ES has resilience to disconnects, frequent reconnects, but in general we assume


resilience

I don't think we are particularly resilient to frequent reconnects. At least, not without being more specific. ES never behaves incorrectly (e.g. loses data) in the face of network outages but it may become unavailable unless the network is stable.

DaveCTurner · 2025-11-15T08:10:26Z

docs/internal/DistributedArchitectureGuide.md

+
+Another area of different networking clients is snapshotting. ES supports
+snapshotting to remote repositories. These repositories usually come with their
+own SDK and networking stack. For example AWS SDK comes with tomcat or netty,


Although the Apache client we use may have originally come from Tomcat, it's not called that any more and this'll confuse folks that don't know the history. Let's just call it the Apache client.

Apache indeed, no idea why I wrote Tomcat.

DaveCTurner · 2025-11-15T08:11:59Z

docs/internal/DistributedArchitectureGuide.md

+be forked to another thread pool. But forking comes with overhead, doing forking
+on every tiny request is a wasted CPU work. As a rule of thumb: don't fork
+simple requests that can be served from memory and do not require heavy
+computations (seconds), otherwise fork.


A pause of "seconds" is still pretty disastrous

Suggested change

computations (seconds), otherwise fork.

computations (millseconds), otherwise fork.

DaveCTurner · 2025-11-15T08:14:12Z

docs/internal/DistributedArchitectureGuide.md

+One of the performance edges of netty is controlled memory allocation. Netty
+manages byte buffer pools and reuse them heavily. This performance gain comes
+with a cost. And cost is reference counting on developer's shoulders. Netty
+reads socket bytes into pooled byte-buffers and passes them to application. Then


Might be worth saying a bit more about how this looks in a heap dump - there's a pool of 1MiB byte[] objects which Netty slices into 16kiB pages, not all of which may be in use, so you have to account for this when investigating memory usage.

DaveCTurner · 2025-11-15T08:15:50Z

docs/internal/DistributedArchitectureGuide.md

+One of the performance edges of netty is controlled memory allocation. Netty
+manages byte buffer pools and reuse them heavily. This performance gain comes
+with a cost. And cost is reference counting on developer's shoulders. Netty
+reads socket bytes into pooled byte-buffers and passes them to application. Then


Also Netty technically reads the bytes into a direct buffer and then CopyBytesSocketChannel copies them into the pooled buffer.

DaveCTurner · 2025-11-15T08:16:57Z

docs/internal/DistributedArchitectureGuide.md

+Another area of different networking clients is snapshotting. ES supports
+snapshotting to remote repositories. These repositories usually come with their
+own SDK and networking stack. For example AWS SDK comes with tomcat or netty,
+Azure with netty-based project-reactor, GCP uses default java HTTP


Netty needs a capital N (here and in several other places)

ywangd

I have some minor comments. Will leave the approval to David. Thanks!

ywangd · 2025-11-16T23:41:05Z

docs/internal/DistributedArchitectureGuide.md

+Security is not enabled by default, meaning no TLS and authentication. These
+features are available in the x-pack/security module.


I would not say "Security is not enabled by default" but instead just say "security features are achieved with separate x-pack modules". We had dedicated effort to ensure security is on by default and basic security including authc/authz/TLS is in the free tier. It is rather disheartening to say the opposite.

ywangd · 2025-11-17T00:02:23Z

docs/internal/DistributedArchitectureGuide.md

+Once node discovers cluster it will open a pool of connections to every other
+node in a cluster, and every other node will open a pool of connections
+back. That means a connection between nodes A and B is A->B pool + B->A pool. A
+node sends requests only on connections it opens (as a client).
+
+A default connection pool is around 13 connections, a pool has sub-pools of
+connections for different purposes: ping, node-state, bulks, etc. Pool structure
+is defined in `ConnectionProfile` class.


I wonder whether we should align the wording here with the code where Conneciton is a higher level abstraction that encapsulates a pool of Channels. Similarly, ConnectionProfile is at higher level while transport profile is for each channel. The most prominent Connection implementation is NodeChannels (note the plura form). It took a while to get my head around them when I first read the code.

Agree with aligning terminology. "Connection is a pool of Channels" might need another explanation that Channel is a non-blocking TCP connection based on Java NIO definition.

ywangd · 2025-11-17T00:13:04Z

docs/internal/DistributedArchitectureGuide.md

+
+## Snapshots
+
+Another area of different networking clients is snapshotting. ES supports


There are other features such as SAML/JWT metadata reloading, watcher http action, reindex and probably ML related features such as inference that also use HTTP clients. I wonder whether we should mention them here. Or do you see snapshots being the most relevant one for the team?

It would be nice to mention all networking interactions and high level overview. To display big picture of ins and outs. I will add ones you mentioned, but will leave details for next round of docs improvement.

mhl-b · 2025-11-18T23:40:06Z

@DaveCTurner, @ywangd

I addressed all comments in new revision.b691644

github-actions · 2025-11-18T23:40:41Z

🔍 Preview links for changed docs

docs/internal/DistributedArchitectureGuide.md

github-actions · 2025-11-18T23:40:42Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

DaveCTurner

Looks good, just a few nits.

DaveCTurner · 2025-11-19T06:32:58Z

docs/internal/DistributedArchitectureGuide.md

+## HTTP Transport
+The HTTP Transport Server (simply HTTP Transport) is a single entry point for


We typically use the word "transport" to mean the node-to-node protocol, to distinguish it from the HTTP network interface. So yes although "HTTP Transport" is kinda correct, it is confusing in this context. Could we avoid using "transport" here?

DaveCTurner · 2025-11-19T06:33:21Z

docs/internal/DistributedArchitectureGuide.md

+by middle boxes.
+
+There is no connection limit, but a limit on payload size exists. The default
+maximum payload is 100MB after compression. It's a very large number and almost


Maybe mention the relevant setting here?

DaveCTurner · 2025-11-19T06:34:07Z

docs/internal/DistributedArchitectureGuide.md

+chunks. It's more complicated for application code, but provides better control
+over memory usage.
+
+Incremental bulk indexing includes a back-pressure feature.When memory pressure


Suggested change

Incremental bulk indexing includes a back-pressure feature.When memory pressure

Incremental bulk indexing includes a back-pressure feature. When memory pressure

DaveCTurner · 2025-11-19T06:34:42Z

docs/internal/DistributedArchitectureGuide.md

+chunks. It's more complicated for application code, but provides better control
+over memory usage.
+
+Incremental bulk indexing includes a back-pressure feature.When memory pressure


Could we link to the org.elasticsearch.index.IndexingPressure here?

DaveCTurner · 2025-11-19T06:35:42Z

docs/internal/DistributedArchitectureGuide.md

+ES supports multiple `Content-Types` for the payload, collectively referred to
+internally as `XContentType`: CBOR, JSON, SMILE, YAML, and their versioned types.


We can also handle other content-types, it's just that these are the best-supported ones.

DaveCTurner · 2025-11-19T06:35:58Z

docs/internal/DistributedArchitectureGuide.md

+
+HTTP routing is based on a combination of Method and URI. For example,
+`RestCreateIndexAction` handler uses `("PUT", "/{index}")`, where curly braces
+indicate path variables. RestBulkAction specifies a list of routes


Suggested change

indicate path variables. RestBulkAction specifies a list of routes

indicate path variables. `RestBulkAction` specifies a list of routes

DaveCTurner · 2025-11-19T06:37:44Z

docs/internal/DistributedArchitectureGuide.md

+servers and clients, handling potentially hundreds or thousands of connections.
+
+Event-loop threads serve many connections each, it's critical to not block
+threads for a long time.Fork any blocking operation or heavy computation to


Suggested change

threads for a long time.Fork any blocking operation or heavy computation to

threads for a long time. Fork any blocking operation or heavy computation to

burqen

I left some suggestions for improvement as I found them while reading.

In the GeneralArchitectureGuide there is a section about # REST and Transport Layers and there is a bit of overlap between that section and what you add here. I think it would be a good idea to combine them into one thing but I also realize that it would be quite a bit of additional work to do so. I leave it as a suggestion to be dealt with either now or in the Glorious Future™.

burqen · 2025-11-19T08:27:29Z

docs/internal/DistributedArchitectureGuide.md

+
+HTTP transport provides two options for content processing: aggregate fully and
+incremental. Aggregated content is a preferable choice for small messages that
+do not fit for incremental parsing (e.g., JSON). Aggregation has drawbacks, it


Suggested change

do not fit for incremental parsing (e.g., JSON). Aggregation has drawbacks, it

are not fit for incremental parsing (e.g., JSON). Aggregation has drawbacks, it

burqen · 2025-11-19T08:28:22Z

docs/internal/DistributedArchitectureGuide.md

+maximum payload is 100MB after compression. It's a very large number and almost
+never a good target that the client should approach.
+
+Security features, including basic security (authc/authz/TLS) in the free tier,


Suggested change

Security features, including basic security (authc/authz/TLS) in the free tier,

Security features, including basic security (authentication/authorization/Transport Layer Security) in the free tier,

For someone that is not used to working with networking and security (me) it's not obvious what authc/authz/TLS means without googling. If target audience is newcomers and open source community I think it's worth spelling those out.

burqen · 2025-11-19T08:32:29Z

docs/internal/DistributedArchitectureGuide.md

+Large delimited content, such as bulk indexing, which is processed in byte
+chunks. It's more complicated for application code, but provides better control
+over memory usage.


Suggested change

Large delimited content, such as bulk indexing, which is processed in byte

chunks. It's more complicated for application code, but provides better control

over memory usage.

Large delimited content, such as bulk indexing, which is processed in byte

chunks, provides better control over memory usage but is more complicated

for application code.

burqen · 2025-11-19T08:37:35Z

docs/internal/DistributedArchitectureGuide.md

+
+`Netty4HttpServerTransport` is a single implementation of
+`AbstractHttpServerTransport` from the `transport-netty4`
+module. Security module injects SSL and headers validator.


Suggested change

module. Security module injects SSL and headers validator.

module. Security module injects SSL (Secure Sockets Layer)

and headers validator.

Not obvious abbreviation for me.

Let's call it TLS rather than SSL - SSL has been deprecated for over a decade at this point

mhl-b · 2025-11-19T18:21:51Z

Thanks for the feedback!

In the GeneralArchitectureGuide there is a section about # REST and Transport Layers and there is a bit of overlap between that section and what you add here.

I think overlap is ok, as long it's consistent. It also overlaps with public docs https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/networking-settings. I will add this to the doc too.

For someone that is not used to working with networking and security

Sometimes I wonder if my life would be better had I never learned computer networks.

mhl-b · 2025-11-19T20:52:06Z

@DaveCTurner, @burqen. Addressed feedback in 08991aa

ywangd

LGTM

I have only optional editorial level comments.

ywangd · 2025-11-20T01:11:29Z

docs/internal/DistributedArchitectureGuide.md

+## HTTP Server
+
+The HTTP Server is a single entry point for all external clients (excluding
+cross-cluster communication). Management,ingestion, search, and all other


Nit

Suggested change

cross-cluster communication). Management,ingestion, search, and all other

cross-cluster communication). Management, ingestion, search, and all other

ywangd · 2025-11-20T01:19:57Z

docs/internal/DistributedArchitectureGuide.md

+This mechanism protects against unbounded memory usage and `OutOfMemory`
+errors (OOMs).
+
+ES supports multiple `Content-Types` for the payload. These are


Nit

Suggested change

ES supports multiple `Content-Types` for the payload. These are

ES supports multiple `Content-Type`s for the payload. These are

?

ywangd · 2025-11-20T01:21:43Z

docs/internal/DistributedArchitectureGuide.md

+implementations of `MediaType` interface. A common ones internally called
+`XContentType`: CBOR, JSON, SMILE, YAML, and their versioned types. X-pack


Nit

Suggested change

implementations of `MediaType` interface. A common ones internally called

`XContentType`: CBOR, JSON, SMILE, YAML, and their versioned types. X-pack

implementations of `MediaType` interface. A common implementation is called

`XContentType`, including CBOR, JSON, SMILE, YAML, and their versioned types. X-pack

ywangd · 2025-11-20T01:23:04Z

docs/internal/DistributedArchitectureGuide.md

+implementations of `MediaType` interface. A common ones internally called
+`XContentType`: CBOR, JSON, SMILE, YAML, and their versioned types. X-pack
+extensions includes PLAIN_TEXT, CSV, etc. Classes that implement
+`ToXContent...` can be serialized and sent over HTTP.


Nit

Suggested change

`ToXContent...` can be serialized and sent over HTTP.

`ToXContent` and friends can be serialized and sent over HTTP.

ywangd · 2025-11-20T01:24:35Z

docs/internal/DistributedArchitectureGuide.md

+
+`Netty4HttpServerTransport` is a single implementation of
+`AbstractHttpServerTransport` from the `transport-netty4`
+module. Security module injects TLS and headers validator.


Suggested change

module. Security module injects TLS and headers validator.

module. The `x-pack/security` module injects TLS and headers validator.

ywangd · 2025-11-20T01:27:59Z

docs/internal/DistributedArchitectureGuide.md

+`Connections` between any two nodes `(A→B and B→A)`. A node sends requests only
+on the `Connection` it opens (acting as a client). The default pool is around 13
+`Channels`, divided into sub-pools for different purposes (e.g., ping,


Not sure if there is a convention here. I personally prefer Connections over Connections since literal string seems to indicate corresponding java concept, e.g. class.

ywangd · 2025-11-20T01:33:21Z

docs/internal/DistributedArchitectureGuide.md

+The compiler does not help detect these issues. They require careful testing
+using Netty's LeakDetector with a Paranoid level.


I think it's enabled by default for tests? If so, maybe worth to mention.

burqen · 2025-11-20T15:57:21Z

Things are looking good, approving 😄

…o networking-guide

mhl-b · 2025-11-22T17:22:48Z

@DaveCTurner, are you ok with current version?

add distrib networking guide docs

a83e71f

elasticsearchmachine added needs:triage Requires assignment of a team area label v9.3.0 labels Nov 14, 2025

github-actions bot deployed to docs-preview November 14, 2025 18:23 View deployment

mhl-b requested review from DaveCTurner and ywangd November 14, 2025 18:25

corrections

78e7752

github-actions bot deployed to docs-preview November 14, 2025 18:55 View deployment

DaveCTurner reviewed Nov 15, 2025

View reviewed changes

ywangd reviewed Nov 17, 2025

View reviewed changes

mhl-b added 2 commits November 18, 2025 15:38

Merge remote-tracking branch 'upstream/main' into networking-guide

19c327e

refine

b691644

github-actions bot deployed to docs-preview November 18, 2025 23:39 View deployment

mhl-b requested review from DaveCTurner and ywangd November 18, 2025 23:40

DaveCTurner reviewed Nov 19, 2025

View reviewed changes

burqen reviewed Nov 19, 2025

View reviewed changes

feedback

08991aa

github-actions bot deployed to docs-preview November 19, 2025 20:51 View deployment

mhl-b requested a review from DaveCTurner November 19, 2025 20:52

mhl-b requested a review from burqen November 19, 2025 20:52

Merge branch 'main' into networking-guide

adee443

github-actions bot deployed to docs-preview November 19, 2025 22:52 View deployment

ywangd approved these changes Nov 20, 2025

View reviewed changes

burqen approved these changes Nov 20, 2025

View reviewed changes

mhl-b added 3 commits November 21, 2025 16:00

update

7b40a82

Merge branch 'networking-guide' of github.com:mhl-b/elasticsearch int…

dd2e60b

…o networking-guide

Merge branch 'main' into networking-guide

0c0895f

github-actions bot deployed to docs-preview November 22, 2025 00:02 View deployment

		be HTTP spec compliant, Elastic is not a webserver. We support GET requests with
		payload (some old proxies might drop content), requests cannot be cached by

		The job of the REST handler is to parse and validate HTTP request and construct
		typed version of request, often Transport request (see Transport section below).

	computations (seconds), otherwise fork.
	computations (millseconds), otherwise fork.

		Security is not enabled by default, meaning no TLS and authentication. These
		features are available in the x-pack/security module.


		## Snapshots

		Another area of different networking clients is snapshotting. ES supports

		## HTTP Transport
		The HTTP Transport Server (simply HTTP Transport) is a single entry point for

	Incremental bulk indexing includes a back-pressure feature.When memory pressure
	Incremental bulk indexing includes a back-pressure feature. When memory pressure

		ES supports multiple `Content-Types` for the payload, collectively referred to
		internally as `XContentType`: CBOR, JSON, SMILE, YAML, and their versioned types.

	indicate path variables. RestBulkAction specifies a list of routes
	indicate path variables. `RestBulkAction` specifies a list of routes

	threads for a long time.Fork any blocking operation or heavy computation to
	threads for a long time. Fork any blocking operation or heavy computation to

	do not fit for incremental parsing (e.g., JSON). Aggregation has drawbacks, it
	are not fit for incremental parsing (e.g., JSON). Aggregation has drawbacks, it

	Security features, including basic security (authc/authz/TLS) in the free tier,
	Security features, including basic security (authentication/authorization/Transport Layer Security) in the free tier,

	module. Security module injects SSL and headers validator.
	module. Security module injects SSL (Secure Sockets Layer)
	and headers validator.

	cross-cluster communication). Management,ingestion, search, and all other
	cross-cluster communication). Management, ingestion, search, and all other

	ES supports multiple `Content-Types` for the payload. These are
	ES supports multiple `Content-Type`s for the payload. These are

		implementations of `MediaType` interface. A common ones internally called
		`XContentType`: CBOR, JSON, SMILE, YAML, and their versioned types. X-pack

	`ToXContent...` can be serialized and sent over HTTP.
	`ToXContent` and friends can be serialized and sent over HTTP.

	module. Security module injects TLS and headers validator.
	module. The `x-pack/security` module injects TLS and headers validator.

		The compiler does not help detect these issues. They require careful testing
		using Netty's LeakDetector with a Paranoid level.

Add networking guide docs #138119

Are you sure you want to change the base?

Add networking guide docs #138119

Conversation

mhl-b commented Nov 14, 2025

Uh oh!

elasticsearchmachine commented Nov 14, 2025

Uh oh!

DaveCTurner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ywangd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhl-b commented Nov 18, 2025

Uh oh!

github-actions bot commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

github-actions bot commented Nov 18, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

DaveCTurner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

burqen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 18, 2025 •

edited

Loading