You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The dataset writer now correctly applies backpressure. However, that backpressure is only applied when the write calls slow down. This only happens when the OS disk cache fills up.
However, filling up the OS disk cache is undesirable. It will cause all running processes to get swapped (assuming the system has any swap configured) and will make the system unusable for anything else.
This typically has no actual benefit to the dataset write. The marginal performance boost provided by the extra RAM is often not worth the cost.
One way to do this would be to use direct I/O (although that comes with a plethora of warnings). Another way might be to flag the output was WONTNEED but I don't know for sure if this works (the OS might still cache it so that it can satisfy the write call quickly). Another way might be to somehow track how much disk cache is being used for writes but that would get complex. I'm sure there are other ways I'm just not aware of yet.
Weston Pace / @westonpace:
Just jotting this down but I did a bit of looking into this and I think a combination of writing the file with O_DSYNC and then using WONTNEED on the written data should accomplish this for Linux, though there may be other ways.
On windows there is a flag FILE_FLAG_WRITE_THROUGH which will be key I think
The dataset writer now correctly applies backpressure. However, that backpressure is only applied when the write calls slow down. This only happens when the OS disk cache fills up.
However, filling up the OS disk cache is undesirable. It will cause all running processes to get swapped (assuming the system has any swap configured) and will make the system unusable for anything else.
This typically has no actual benefit to the dataset write. The marginal performance boost provided by the extra RAM is often not worth the cost.
One way to do this would be to use direct I/O (although that comes with a plethora of warnings). Another way might be to flag the output was WONTNEED but I don't know for sure if this works (the OS might still cache it so that it can satisfy the write call quickly). Another way might be to somehow track how much disk cache is being used for writes but that would get complex. I'm sure there are other ways I'm just not aware of yet.
Reporter: Weston Pace / @westonpace
Assignee: Ziheng Wang / @marsupialtail
Related issues:
PRs and other links:
Note: This issue was originally created as ARROW-14635. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: