You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
About 30% faster with plugins: the filters, the plugin calls and the rows of a batch run
on a thread of their own while the next batch is read, as without plugins, with the same
Parquet bytes as before. On a 2019 laptop, a 1 GB tall CSV with the examples/rates
config 7.7 -> 5.2 seconds, the 2.5 GB wide CSV 5.8 -> 4.0, a 229 MB JSON 2.1 -> 1.5.
With plugins, a batch now ends early where the file's metadata changes, so each record
has the metadata it was read with, and a parse error still has the records before it
written.