docs(inkless): add az-alignment feature documentation#481
docs(inkless): add az-alignment feature documentation#481giuseppelillo merged 3 commits intomainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds comprehensive documentation for the client rack awareness feature in Inkless, which routes client requests to brokers in the same availability zone to minimize cross-AZ data transfer costs for diskless topics.
Changes:
- Created new
CLIENT-BROKER-AZ-ALIGNMENT.mddocumentation covering currentclient.idpattern implementation and future KIP-1123 support - Updated FAQ.md to replace detailed rack awareness explanations with references to the new documentation
- Documented configuration, monitoring, troubleshooting, and migration path
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| docs/inkless/CLIENT-BROKER-AZ-ALIGNMENT.md | New comprehensive documentation for client rack awareness feature with configuration examples, deployment diagrams, and migration guidance |
| docs/inkless/FAQ.md | Updated rack awareness Q&A entries to reference new documentation instead of inline explanations |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add comprehensive documentation for client rack awareness feature that routes client requests to brokers in the same availability zone, reducing cross-AZ data transfer costs for diskless topics. - Document current implementation using `client.id` pattern with `diskless_az=<rack>` marker - Explain relationship to existing Kafka rack awareness features - Cover future KIP-1123 support for standard `client.rack` config - Include configuration guide, monitoring metrics, and troubleshooting - Update FAQ with references to new docs
jeqo
force-pushed
the
jeqo/docs-az-alignment
branch
from
c626eae to
bce9fa0
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| 1. **Producers** send produce requests to brokers in their local AZ | ||
| 2. **Consumers** fetch data from brokers in their local AZ (preferring local cached data) | ||
| 3. **Brokers** store data in shared object storage accessible from all AZs | ||
| 4. **Caching** is AZ-local, with each AZ maintaining its own cache |
There was a problem hiding this comment.
Am I missing something here? Caching is now local to the broker, not AZ-local
There was a problem hiding this comment.
it's AZ-local as in entries are cached once per AZ; but yes, we could clarified that is not a distributed cache as before if that is unclear.
| 1. Producer sends Produce request with `client.id=my-producer,diskless_az=us-east-1a` | ||
| 2. Broker extracts AZ: `us-east-1a` | ||
| 3. Broker compares with its own `broker.rack` configuration | ||
| 4. If match, broker accepts and processes the request locally | ||
| 5. If no match, broker can still process but with cross-AZ costs |
There was a problem hiding this comment.
Is this correct to mention? AZ routing does not happen in the broker during produce request handling
There was a problem hiding this comment.
Ah good point, we should make it clearer that it happens on the metadata request and not on the produce one.
| 4. If match, broker accepts and processes the request locally | ||
| 5. If no match, broker can still process but with cross-AZ costs | ||
|
|
||
| **Metadata Response Behavior:** |
There was a problem hiding this comment.
Stylistic note: using bold is visually almost the same as using a heading 4 (at least with the GitHub UI). Example:
So it seems that Metadata Response Behavior is a different paragraph than Producer Flow. Maybe avoiding using the bold helps with the reading clarity.
|
|
||
| **Key Points:** | ||
| - If no brokers exist in the client's rack, brokers from other racks are used as fallback | ||
| - Cache is AZ-local; each rack has its own distributed cache |
There was a problem hiding this comment.
This is also not true anymore
There was a problem hiding this comment.
Good catch, will clarify.
| │ │ az=us-east-1a │ │ az=us-east-1b │ │ s_az=us │ │ | ||
| │ │ │ │ │ │ -east-1c│ │ | ||
| │ │ │ │ │ │ │ │ | ||
| │ │ Cache (local) │ │ Cache (local) │ │ Cache │ │ |
There was a problem hiding this comment.
This cache looks AZ local but it's not
* docs(inkless): add az-alignment feature documentation Add comprehensive documentation for client rack awareness feature that routes client requests to brokers in the same availability zone, reducing cross-AZ data transfer costs for diskless topics. - Document current implementation using `client.id` pattern with `diskless_az=<rack>` marker - Explain relationship to existing Kafka rack awareness features - Cover future KIP-1123 support for standard `client.rack` config - Include configuration guide, monitoring metrics, and troubleshooting - Update FAQ with references to new docs * fixup! docs(inkless): add az-alignment feature documentation * fixup! docs(inkless): add az-alignment feature documentation
* docs(inkless): add az-alignment feature documentation Add comprehensive documentation for client rack awareness feature that routes client requests to brokers in the same availability zone, reducing cross-AZ data transfer costs for diskless topics. - Document current implementation using `client.id` pattern with `diskless_az=<rack>` marker - Explain relationship to existing Kafka rack awareness features - Cover future KIP-1123 support for standard `client.rack` config - Include configuration guide, monitoring metrics, and troubleshooting - Update FAQ with references to new docs * fixup! docs(inkless): add az-alignment feature documentation * fixup! docs(inkless): add az-alignment feature documentation
Add comprehensive documentation for client rack awareness feature that routes client requests to brokers in the same availability zone, reducing cross-AZ data transfer costs for diskless topics.
client.idpattern withdiskless_az=<rack>markerclient.rackconfig