You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
The file-based export uses the COS properties for checkpointing.
This is a bit confusing and, additionally, I think we'd want a different default for "when to start writing" (in COS there are
constraints on the size and count of the parts, while writing to local disk has different concerns).
Describe the solution you'd like
A better default for when to start writing a file. I was thinking it should just checkpoint at every item (i.e. each page or results), but Paul suggested this could contribute to fragmentation. Therefor, lets introduce a new config param for it and set it to 1MB as the default.
Describe alternatives you've considered
A new "fast" implementation that uses nio to asynchronously queue the bytes for the OS / disk controller (instead of synchronously writing chunks to the file). I still think that would be a better implementation, but its more than I want to chew off right now.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
The file-based export uses the COS properties for checkpointing.
This is a bit confusing and, additionally, I think we'd want a different default for "when to start writing" (in COS there are
constraints on the size and count of the parts, while writing to local disk has different concerns).
Describe the solution you'd like
A better default for when to start writing a file. I was thinking it should just checkpoint at every item (i.e. each page or results), but Paul suggested this could contribute to fragmentation. Therefor, lets introduce a new config param for it and set it to 1MB as the default.
Describe alternatives you've considered
A new "fast" implementation that uses nio to asynchronously queue the bytes for the OS / disk controller (instead of synchronously writing chunks to the file). I still think that would be a better implementation, but its more than I want to chew off right now.
The text was updated successfully, but these errors were encountered: