From 3188aa2447cee69bc305486d716defd92e0bdcf3 Mon Sep 17 00:00:00 2001 From: Will Jones Date: Fri, 17 Jul 2026 11:13:05 -0700 Subject: [PATCH 1/5] docs: rewrite write performance/ingestion section Reworks the Ingestion section on the Performance Tips page: - Frames write_parallelism around its main benefit (bandwidth), not just memory, and documents the progress=True tqdm bar (throughput and active worker count) that was previously undocumented. - Clarifies that for larger-than-memory data, scanning a file-backed Dataset is preferable to a hand-built RecordBatchReader: only a Dataset can be counted and rescanned, which enables auto-sized parallelism and retry-on-failure. - Adds write_parallelism guidance for iterator ingestion, where LanceDB can't auto-size it, including a rule of thumb to avoid over- fragmenting small datasets. Alternative to #312, which framed write_parallelism as primarily a memory-capping knob and didn't mention the progress bar. --- docs/performance.mdx | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/docs/performance.mdx b/docs/performance.mdx index 59053411..750c4289 100644 --- a/docs/performance.mdx +++ b/docs/performance.mdx @@ -34,7 +34,7 @@ Use `add()` for pure appends, and reach for `merge_insert()` only when you need ### Bulk ingestion: for data you already have -For materialized inputs (Arrow Tables, DataFrames, `pyarrow.dataset(...)`), LanceDB auto-parallelizes the write across workers. +For materialized inputs (Arrow Tables, DataFrames) and file-backed sources (`pyarrow.dataset(...)`), LanceDB auto-parallelizes the write across workers, estimating the partition count from the data size. The main benefit is write bandwidth: partitions write concurrently, so more of them means more throughput, up to the number of CPU cores available. ```python Python icon="python" table.add(arrow_table) # in-memory @@ -42,11 +42,25 @@ table.add(df) # pandas table.add(ds.dataset("data/", format="parquet")) # streams from disk, still parallelized ``` -`pa.dataset(...)` is the right choice for large file-based loads: it streams Parquet/CSV without loading into memory, and still preserves auto-parallelism. For very large initial loads, create the table empty first; passing data directly to `create_table(name, data)` skips the auto-parallel path. +Pass `progress=True` to watch it happen: LanceDB displays a live tqdm bar with rows written, write throughput in MB/s, and how many of the available workers are active. + +```python Python icon="python" +table.add(ds.dataset("data/", format="parquet"), progress=True) +``` + +For larger-than-memory data, prefer scanning a file-backed dataset (`ds.dataset(...)`) over building your own `pyarrow.RecordBatchReader`. Both stream without loading the whole table into memory, but only a `Dataset` can be counted and rescanned: LanceDB knows the total row count upfront, which lets it auto-size parallelism, and it can retry a failed write by rescanning from the start. A hand-built reader can only be consumed once, so neither is possible — reach for the iterator path below only when the data genuinely can't be backed by files, e.g. embeddings computed on the fly. + +For very large initial loads, create the table empty first; passing data directly to `create_table(name, data)` skips the auto-parallel path. + + +**Wide rows and scan memory** + +Scanning a `pyarrow.dataset.Dataset` or `LanceDataset` for `add()` automatically bounds the scanner's batch size and read-ahead based on the schema's bytes-per-row, so wide-row datasets (large embedding columns, long text, many columns) stay near a ~1 GiB in-flight budget during the scan. Narrow datasets keep PyArrow's defaults, so throughput is unaffected. + ### Iterator ingestion: for streaming or computed-on-the-fly data -If your dataset doesn't fit in memory all at once (for example, when rows are computed on the fly (generating embeddings as you ingest), or pulled from a streaming source, materializing the full table before calling `add()` will cause you to run out of memory. The best practice is to pass an iterator of `pyarrow.RecordBatch` instead. LanceDB consumes one batch at a time, so peak memory stays bounded by the batch size you yield. +If your dataset doesn't fit in memory all at once — rows computed on the fly (generating embeddings as you ingest), or pulled from a streaming source — materializing the full table before calling `add()` will run you out of memory. The best practice is to pass an iterator of `pyarrow.RecordBatch` instead. LanceDB consumes one batch at a time, so peak memory stays bounded by the batch size you yield. ```python Python icon="python" def stream(): @@ -57,7 +71,13 @@ def stream(): table.add(stream()) ``` -Streaming inputs don't auto-parallelize, so use decently large chunks of several thousand rows or more, rather than yielding single-row batches. +Use decently large chunks of several thousand rows or more, rather than yielding single-row batches. Unlike a `Dataset`, a reader can't be counted ahead of time, so LanceDB can't auto-size parallelism for it — set `write_parallelism` yourself: + +```python Python icon="python" +table.add(stream(), write_parallelism=4) +``` + +More parallelism means more write bandwidth, but each partition also becomes its own fragment, so over-parallelizing a small stream just fragments the table for no benefit. As a rule of thumb, don't budget more than one unit of parallelism per ~100K rows or ~1 GB of data — a 1,000-row stream needs `write_parallelism=1`, not more. `write_parallelism` is also a memory knob independent of that tradeoff: each partition buffers a batch in flight, so lower it if ingestion is using more memory than expected. ## Indexing From 72f733af1953ddbb3d0fe914a65956cff0473e76 Mon Sep 17 00:00:00 2001 From: Will Jones Date: Fri, 17 Jul 2026 11:24:40 -0700 Subject: [PATCH 2/5] docs: show sample tqdm output for progress=True --- docs/performance.mdx | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/performance.mdx b/docs/performance.mdx index 750c4289..3f51476b 100644 --- a/docs/performance.mdx +++ b/docs/performance.mdx @@ -48,6 +48,10 @@ Pass `progress=True` to watch it happen: LanceDB displays a live tqdm bar with r table.add(ds.dataset("data/", format="parquet"), progress=True) ``` +```text +71%|███████▏ | 710000/1000000 [00:12<00:05, 58.8kit/s, 42.3 MB/s | 8/8 workers] +``` + For larger-than-memory data, prefer scanning a file-backed dataset (`ds.dataset(...)`) over building your own `pyarrow.RecordBatchReader`. Both stream without loading the whole table into memory, but only a `Dataset` can be counted and rescanned: LanceDB knows the total row count upfront, which lets it auto-size parallelism, and it can retry a failed write by rescanning from the start. A hand-built reader can only be consumed once, so neither is possible — reach for the iterator path below only when the data genuinely can't be backed by files, e.g. embeddings computed on the fly. For very large initial loads, create the table empty first; passing data directly to `create_table(name, data)` skips the auto-parallel path. From 43e83b404052ad06be1bafd950067379fe07cf72 Mon Sep 17 00:00:00 2001 From: Will Jones Date: Fri, 17 Jul 2026 13:04:16 -0700 Subject: [PATCH 3/5] docs: narrow iterator ingestion to on-the-fly transforms Drops the memory-bound framing (iterator ingestion isn't really about memory) and scopes the section to data that needs per-row work before writing, rather than streaming sources generally. Calls out setting write_parallelism manually for large inputs in its own note. --- docs/performance.mdx | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/performance.mdx b/docs/performance.mdx index 3f51476b..affb397a 100644 --- a/docs/performance.mdx +++ b/docs/performance.mdx @@ -62,9 +62,9 @@ For very large initial loads, create the table empty first; passing data directl Scanning a `pyarrow.dataset.Dataset` or `LanceDataset` for `add()` automatically bounds the scanner's batch size and read-ahead based on the schema's bytes-per-row, so wide-row datasets (large embedding columns, long text, many columns) stay near a ~1 GiB in-flight budget during the scan. Narrow datasets keep PyArrow's defaults, so throughput is unaffected. -### Iterator ingestion: for streaming or computed-on-the-fly data +### Iterator ingestion: for data you transform on the fly -If your dataset doesn't fit in memory all at once — rows computed on the fly (generating embeddings as you ingest), or pulled from a streaming source — materializing the full table before calling `add()` will run you out of memory. The best practice is to pass an iterator of `pyarrow.RecordBatch` instead. LanceDB consumes one batch at a time, so peak memory stays bounded by the batch size you yield. +If each row needs work before it can be written — generating embeddings as you ingest, for example — the data doesn't exist as a file you can point `ds.dataset(...)` at. The best practice is to pass an iterator of `pyarrow.RecordBatch` instead; LanceDB consumes one batch at a time as you produce them. ```python Python icon="python" def stream(): @@ -75,13 +75,19 @@ def stream(): table.add(stream()) ``` -Use decently large chunks of several thousand rows or more, rather than yielding single-row batches. Unlike a `Dataset`, a reader can't be counted ahead of time, so LanceDB can't auto-size parallelism for it — set `write_parallelism` yourself: +Use decently large chunks of several thousand rows or more, rather than yielding single-row batches. + + +**Set `write_parallelism` yourself for large inputs** + +Unlike a `Dataset`, a reader can't be counted or rescanned ahead of time, so LanceDB can't auto-size parallelism for it. For large inputs, set `write_parallelism` explicitly: ```python Python icon="python" table.add(stream(), write_parallelism=4) ``` -More parallelism means more write bandwidth, but each partition also becomes its own fragment, so over-parallelizing a small stream just fragments the table for no benefit. As a rule of thumb, don't budget more than one unit of parallelism per ~100K rows or ~1 GB of data — a 1,000-row stream needs `write_parallelism=1`, not more. `write_parallelism` is also a memory knob independent of that tradeoff: each partition buffers a batch in flight, so lower it if ingestion is using more memory than expected. +More parallelism means more write bandwidth, but each partition also becomes its own fragment, so don't over-allocate on a small input — budget one unit of parallelism per ~100K rows or ~1 GB of data as a rule of thumb. + ## Indexing From 9d5ba246b0563f4009fd6a21d0928a53233b40f9 Mon Sep 17 00:00:00 2001 From: Will Jones Date: Fri, 17 Jul 2026 13:08:31 -0700 Subject: [PATCH 4/5] docs: tighten ingestion section prose Trims repeated reasoning (bandwidth benefit, Dataset vs RecordBatchReader rescan/retry logic) that was restated across paragraphs. --- docs/performance.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/performance.mdx b/docs/performance.mdx index affb397a..f093c9b7 100644 --- a/docs/performance.mdx +++ b/docs/performance.mdx @@ -34,7 +34,7 @@ Use `add()` for pure appends, and reach for `merge_insert()` only when you need ### Bulk ingestion: for data you already have -For materialized inputs (Arrow Tables, DataFrames) and file-backed sources (`pyarrow.dataset(...)`), LanceDB auto-parallelizes the write across workers, estimating the partition count from the data size. The main benefit is write bandwidth: partitions write concurrently, so more of them means more throughput, up to the number of CPU cores available. +For materialized inputs (Arrow Tables, DataFrames) and file-backed sources (`pyarrow.dataset(...)`), LanceDB auto-parallelizes the write across workers, estimating the partition count from the data size — more partitions means more concurrent writes and higher throughput, up to the CPU core count. ```python Python icon="python" table.add(arrow_table) # in-memory @@ -42,7 +42,7 @@ table.add(df) # pandas table.add(ds.dataset("data/", format="parquet")) # streams from disk, still parallelized ``` -Pass `progress=True` to watch it happen: LanceDB displays a live tqdm bar with rows written, write throughput in MB/s, and how many of the available workers are active. +Pass `progress=True` to watch it happen: LanceDB shows a live tqdm bar with rows written, throughput in MB/s, and active worker count. ```python Python icon="python" table.add(ds.dataset("data/", format="parquet"), progress=True) @@ -52,7 +52,7 @@ table.add(ds.dataset("data/", format="parquet"), progress=True) 71%|███████▏ | 710000/1000000 [00:12<00:05, 58.8kit/s, 42.3 MB/s | 8/8 workers] ``` -For larger-than-memory data, prefer scanning a file-backed dataset (`ds.dataset(...)`) over building your own `pyarrow.RecordBatchReader`. Both stream without loading the whole table into memory, but only a `Dataset` can be counted and rescanned: LanceDB knows the total row count upfront, which lets it auto-size parallelism, and it can retry a failed write by rescanning from the start. A hand-built reader can only be consumed once, so neither is possible — reach for the iterator path below only when the data genuinely can't be backed by files, e.g. embeddings computed on the fly. +For larger-than-memory data, prefer scanning a file-backed dataset (`ds.dataset(...)`) over a hand-built `pyarrow.RecordBatchReader`: a `Dataset` can be counted and rescanned, so LanceDB knows the row count upfront (better auto-parallelism) and can retry a failed write from the start — a reader can only be consumed once, so neither is possible. Reach for the iterator path below only when the data genuinely can't be backed by files, e.g. embeddings computed on the fly. For very large initial loads, create the table empty first; passing data directly to `create_table(name, data)` skips the auto-parallel path. @@ -80,13 +80,13 @@ Use decently large chunks of several thousand rows or more, rather than yielding **Set `write_parallelism` yourself for large inputs** -Unlike a `Dataset`, a reader can't be counted or rescanned ahead of time, so LanceDB can't auto-size parallelism for it. For large inputs, set `write_parallelism` explicitly: +A reader can't be counted or rescanned the way a `Dataset` can (see above), so LanceDB can't auto-size parallelism for it — set `write_parallelism` explicitly for large inputs: ```python Python icon="python" table.add(stream(), write_parallelism=4) ``` -More parallelism means more write bandwidth, but each partition also becomes its own fragment, so don't over-allocate on a small input — budget one unit of parallelism per ~100K rows or ~1 GB of data as a rule of thumb. +Each partition becomes its own fragment, so don't over-allocate on a small input — budget one unit of parallelism per ~100K rows or ~1 GB of data as a rule of thumb. ## Indexing From 08cb771b2b715398c3f8ddb991d66a6a4c2cc7c1 Mon Sep 17 00:00:00 2001 From: Will Jones Date: Fri, 17 Jul 2026 13:53:50 -0700 Subject: [PATCH 5/5] docs: fix embeddings example, drop wide-row scan memory note Embedding computation is built into the ingestion pipeline, so it's a misleading example of when you'd need to hand-roll an iterator instead of scanning a file-backed Dataset. Swaps it for a generic "custom data transformations" example in both places it appeared. Also drops the wide-row scan memory bounding note for now. --- docs/performance.mdx | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/docs/performance.mdx b/docs/performance.mdx index f093c9b7..e6c6be02 100644 --- a/docs/performance.mdx +++ b/docs/performance.mdx @@ -52,19 +52,13 @@ table.add(ds.dataset("data/", format="parquet"), progress=True) 71%|███████▏ | 710000/1000000 [00:12<00:05, 58.8kit/s, 42.3 MB/s | 8/8 workers] ``` -For larger-than-memory data, prefer scanning a file-backed dataset (`ds.dataset(...)`) over a hand-built `pyarrow.RecordBatchReader`: a `Dataset` can be counted and rescanned, so LanceDB knows the row count upfront (better auto-parallelism) and can retry a failed write from the start — a reader can only be consumed once, so neither is possible. Reach for the iterator path below only when the data genuinely can't be backed by files, e.g. embeddings computed on the fly. +For larger-than-memory data, prefer scanning a file-backed dataset (`ds.dataset(...)`) over a hand-built `pyarrow.RecordBatchReader`: a `Dataset` can be counted and rescanned, so LanceDB knows the row count upfront (better auto-parallelism) and can retry a failed write from the start — a reader can only be consumed once, so neither is possible. Reach for the iterator path below only when the data genuinely can't be backed by files, e.g. to apply custom data transformations as you ingest. For very large initial loads, create the table empty first; passing data directly to `create_table(name, data)` skips the auto-parallel path. - -**Wide rows and scan memory** - -Scanning a `pyarrow.dataset.Dataset` or `LanceDataset` for `add()` automatically bounds the scanner's batch size and read-ahead based on the schema's bytes-per-row, so wide-row datasets (large embedding columns, long text, many columns) stay near a ~1 GiB in-flight budget during the scan. Narrow datasets keep PyArrow's defaults, so throughput is unaffected. - - ### Iterator ingestion: for data you transform on the fly -If each row needs work before it can be written — generating embeddings as you ingest, for example — the data doesn't exist as a file you can point `ds.dataset(...)` at. The best practice is to pass an iterator of `pyarrow.RecordBatch` instead; LanceDB consumes one batch at a time as you produce them. +If each row needs work before it can be written — applying a custom transformation as you ingest, for example — the data doesn't exist as a file you can point `ds.dataset(...)` at. The best practice is to pass an iterator of `pyarrow.RecordBatch` instead; LanceDB consumes one batch at a time as you produce them. ```python Python icon="python" def stream():