You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have seen weird IOErrors on long running ktk/dask computations that have caused incidents
These errors happen while reading Parquet files from Azure blob.
One possible cause for this is kartothek's custom implementation of BlockBuffer. While it was useful at the time of implementation, we can look into replacing this with the pyarrow buffer, so that we don't need to maintain this complex piece of code and can discard this a source for the problem.
We'll want to check potential performance implications of this change.
Problem description
As commented by @NeroCorleone in #397:
These errors happen while reading Parquet files from Azure blob.
One possible cause for this is kartothek's custom implementation of BlockBuffer. While it was useful at the time of implementation, we can look into replacing this with the pyarrow buffer, so that we don't need to maintain this complex piece of code and can discard this a source for the problem.
We'll want to check potential performance implications of this change.
Implementation hint:
fa2af5c
(#397)The text was updated successfully, but these errors were encountered: