Skip to content

Update arch diagram #317

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tsmithv11 merged 2 commits into from
Jul 24, 2026
Merged

Update arch diagram #317

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
065e5da
Update arch diagram
tsmithv11 Jul 22, 2026
d85acf6
Update based on final from eng
tsmithv11 Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 27 additions & 22 deletions docs/enterprise/architecture.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,31 +48,36 @@ Indexers handle heavyweight background work such as building indexes, merging in

```mermaid
flowchart TB
Client[Client application]
Remote[Remote table / API boundary]
Client[Client applications]

subgraph DP[Data plane]
QN[Query nodes]
PE[Plan executors<br/>cache + distributed execution]
IX[Indexers / background workers]
subgraph CP[Control plane]
CFG[Configuration, identity, policy, lifecycle]
end

subgraph CP[Control plane]
CFG[Configuration, identity,<br/>policy, lifecycle]
subgraph DP[Data Plane<br/><span style='font-size:0.85em'>Managed or self-hosted</span>]
QP[Query planning + routing fleet]
PE[Executor + cache fleet]
MON[Monitoring + orchestration fleet]
WAL[MemWAL fleet]
JOBS[Jobs fleet]

QP -->|Query subplans| PE
QP -->|Send state + job requests| MON
QP -->|Fresh reads and writes| WAL
PE -.->|Send state| MON
WAL -.->|Send state| MON
MON -->|Indexing, compaction,<br/>backfill, refresh jobs| JOBS
JOBS -->|Send state| MON
end

OS[(Object storage<br/>table data, manifests, indexes)]

Client -->|Read / write requests| Remote
Remote --> QN
QN -->|Query sub-plans| PE
QN -->|Commits writes| OS
PE <-->|Read data| OS
IX <-->|Build indexes / compact data| OS
QN -.->|Indexing / compaction signals| IX
CFG -.->|Govern and configure| QN
CFG -.->|Govern and configure| PE
CFG -.->|Govern and configure| IX
OS[(Object Storage)]

Client -->|Queries, writes, backfills| DP
CFG -.->|Govern and configure| DP
PE -->|Read, cache data| OS
JOBS -->|Read/write table and feature data| OS
MON -.->|Table and cluster state| OS
WAL -->|Persist| OS
```

## Remote tables
Expand Down Expand Up @@ -111,8 +116,8 @@ Indexing, compaction, and cleanup are intentionally moved off the user request p

In practice, that usually looks like this:

1. Table changes create follow-up signals for indexing, compaction, or cleanup.
2. A background coordinator agent evaluates the state of the table and decides what should run next.
1. Table changes create follow-up events that query nodes publish to an event queue for indexing, compaction, or cleanup.
2. A background coordinator agent consumes those events, evaluates the state of the table, and decides what should run next.
3. Indexers read the relevant table state from object storage, produce updated artifacts, and write the results back.

This separation is one of the clearest architectural reasons to use LanceDB Enterprise: the same query-serving infrastructure does not have to handle every expensive indexing or compaction task itself.
Expand Down
Loading