docs: rewrite write performance/ingestion section

docs/performance.mdx

@@ -34,19 +34,31 @@ Use `add()` for pure appends, and reach for `merge_insert()` only when you need
 
 ### Bulk ingestion: for data you already have
 
-For materialized inputs (Arrow Tables, DataFrames, `pyarrow.dataset(...)`), LanceDB auto-parallelizes the write across workers.
+For materialized inputs (Arrow Tables, DataFrames) and file-backed sources (`pyarrow.dataset(...)`), LanceDB auto-parallelizes the write across workers, estimating the partition count from the data size — more partitions means more concurrent writes and higher throughput, up to the CPU core count.
 
 ```python Python icon="python"
 table.add(arrow_table)                              # in-memory
 table.add(df)                                       # pandas
 table.add(ds.dataset("data/", format="parquet"))    # streams from disk, still parallelized
 ```
 
-`pa.dataset(...)` is the right choice for large file-based loads: it streams Parquet/CSV without loading into memory, and still preserves auto-parallelism. For very large initial loads, create the table empty first; passing data directly to `create_table(name, data)` skips the auto-parallel path.
+Pass `progress=True` to watch it happen: LanceDB shows a live tqdm bar with rows written, throughput in MB/s, and active worker count.
 
-### Iterator ingestion: for streaming or computed-on-the-fly data
+```python Python icon="python"
+table.add(ds.dataset("data/", format="parquet"), progress=True)
+```
+
+```text
+71%|███████▏  | 710000/1000000 [00:12<00:05, 58.8kit/s, 42.3 MB/s | 8/8 workers]
+```
 
-If your dataset doesn't fit in memory all at once (for example, when rows are computed on the fly (generating embeddings as you ingest), or pulled from a streaming source, materializing the full table before calling `add()` will cause you to run out of memory. The best practice is to pass an iterator of `pyarrow.RecordBatch` instead. LanceDB consumes one batch at a time, so peak memory stays bounded by the batch size you yield.
+For larger-than-memory data, prefer scanning a file-backed dataset (`ds.dataset(...)`) over a hand-built `pyarrow.RecordBatchReader`: a `Dataset` can be counted and rescanned, so LanceDB knows the row count upfront (better auto-parallelism) and can retry a failed write from the start — a reader can only be consumed once, so neither is possible. Reach for the iterator path below only when the data genuinely can't be backed by files, e.g. to apply custom data transformations as you ingest.
+
+For very large initial loads, create the table empty first; passing data directly to `create_table(name, data)` skips the auto-parallel path.
+
+### Iterator ingestion: for data you transform on the fly
+
+If each row needs work before it can be written — applying a custom transformation as you ingest, for example — the data doesn't exist as a file you can point `ds.dataset(...)` at. The best practice is to pass an iterator of `pyarrow.RecordBatch` instead; LanceDB consumes one batch at a time as you produce them.
 
 ```python Python icon="python"
 def stream():
@@ -57,7 +69,19 @@ def stream():
 table.add(stream())
 ```
 
-Streaming inputs don't auto-parallelize, so use decently large chunks of several thousand rows or more, rather than yielding single-row batches.
+Use decently large chunks of several thousand rows or more, rather than yielding single-row batches.
+
+<Note>
+**Set `write_parallelism` yourself for large inputs**
+
+A reader can't be counted or rescanned the way a `Dataset` can (see above), so LanceDB can't auto-size parallelism for it — set `write_parallelism` explicitly for large inputs:
+
+```python Python icon="python"
+table.add(stream(), write_parallelism=4)
+```
+
+Each partition becomes its own fragment, so don't over-allocate on a small input — budget one unit of parallelism per ~100K rows or ~1 GB of data as a rule of thumb.
+</Note>
 
 ## Indexing