perf(flow-php/etl-adapter-csv): reuse a single row buffer and append … - #2518
Merged
norberttech merged 2 commits intoJul 11, 2026
Merged
Conversation
…per batch in CSVLoader CSVLoader::writeCSV opened and closed a php://temp stream for every written row and resolved the destination stream through FilesystemStreams::writeTo() inside the per-row loop. At 1M rows that meant ~4M stream-API calls and 1M redundant path validations, making CSV writing more expensive than reading. - keep one lazily-opened php://temp buffer per loader instance, truncated and rewound between uses, released in the existing closure() hook - collect the whole normalized batch in the buffer and append it to the destination stream once per Rows batch instead of once per row - hoist the loop-invariant writeTo() call out of the row loop - add a regression test asserting exact file bytes for rows of decreasing length across multiple batches (guards the ftruncate between buffer reuses) Measured on the 1M-row benchmark dataset: 11.1s -> 7.4s (-34%) with batchSize(1000), byte-identical output (cmp on 59.8MB result).
…le-row-buffer # Conflicts: # src/adapter/etl-adapter-csv/tests/Flow/ETL/Adapter/CSV/Tests/Integration/CSVTest.php
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 1.x #2518 +/- ##
============================================
Coverage 85.96% 85.97%
- Complexity 0 22261 +22261
============================================
Files 1663 1663
Lines 68406 68416 +10
============================================
+ Hits 58807 58818 +11
+ Misses 9599 9598 -1 🚀 New features to boost your workflow:
|
MrHDOLEK
marked this pull request as ready for review
July 11, 2026 10:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CSVLoader::writeCSV()opened and closed aphp://tempstream for every written row and resolved the destination stream throughFilesystemStreams::writeTo()inside theper-row loop. Writing 1M rows meant:
fopen('php://temp')/fputcsv/stream_get_contents/fclose(~4M stream-API calls),writeTo()— each re-running path validation,pathinfo()and apreg_match_allon loop-invariant arguments (visible in profiles asPath->uri()× 300k per 100krows),
DestinationStream::append()— onefwriteper row.The result: writing a CSV was ~80% more expensive than reading the same file. Found while profiling the full pipeline against a 1M-row dataset (Xdebug/cachegrind — the loader
machinery accounted for ~22% of self-time on the write path).
Changes
php://tempbuffer per loader instance, truncated and rewound between uses, released in the existingclosure()hook (Loader\Closure); lazy re-openvia
is_resource()also keeps the loader safe to serialize,Rowsbatch instead of once per row,writeTo()call out of the row loop,ftruncate()between buffer reuses).No public API changes. Output is byte-identical (verified with
cmpon a 59.8 MB / 1M-row result file).Benchmarks
1M-row CSV (57 MB, 2 string columns), PHP 8.3 + opcache, medians of 3 runs per config, each run in a fresh process:
1.xto_csv(),batchSize(1000)to_csv(),batchSize(10000)to_csv(), default batch (1 row)Pure write overhead (write minus read of the same file,
batchSize(1000)): 4.68 s → 0.49 s (~10×) — writing now costs roughly the same as reading (+9%, previously +82%). Afterthe change,
CSVLoader/writeTo/appendno longer appear among the top functions in the cachegrind profile of the write path.Peak memory is unchanged (16 MB at
batchSize(1000); +2 MB atbatchSize(10000)for the batch buffer).The small gain at the default batch size is expected: with 1-row batches every per-batch cost is a per-row cost (
load(), telemetry,writeTo,appendstill run once per row) —that is a pipeline-level batching concern, independent of this loader fix.
Resolves: no related roadmap item — standalone performance fix, happy to open an issue if preferred
Change Log
Added
Fixed
Changed
Removed
Deprecated
Security