-
Notifications
You must be signed in to change notification settings - Fork 103
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Invalid/empty header with write_csv #81
Comments
@hayesgb any pointers on how to debug this? Thanks |
Does repartitioning the dataframe before writing it resolve the problem? Are you still seeing the issue with either 0.4.x or 0.5.0? If so, can you provide me with a small example that replicates the problem? Something like this:
|
@hayesgb repartitioning the dataframe before, with I've tried 0.5.0 briefly, need more time to see if it's working. P.S. - What I notice with 0.5.0 (you're using asyncio correct?) is that some delete operations that were fast before, are really slow sometimes and sometimes really fast. I use Python Interactive in VSCode a lot and sometimes it just gets stuck. As reported by more people, looks like VSCode Python Interactive is not too friendly with asyncio multiple processess. |
v0.5.x uses asyncio and caching for filesystem operation (not for the AzureBlobFile class) to speed up operations. |
@hayesgb quick way to cause this issue: file = open('sample.txt', 'wb') #empty file
abfs.put("./sample.txt", '<blob_name>/sample.txt') Traceback~/miniconda3/envs/pipelines/lib/python3.8/site-packages/adlfs/spec.py in put(self, lpath, rpath, recursive, **kwargs) 1121 1122 for lpath, rpath in zip(lpaths, rpaths): -> 1123 self.put_file(lpath, rpath, **kwargs) 1124 1125 def upload(self, lpath, rpath, recursive=False, **kwargs):~/miniconda3/envs/pipelines/lib/python3.8/site-packages/fsspec/spec.py in put_file(self, lpath, rpath, **kwargs) ~/miniconda3/envs/pipelines/lib/python3.8/site-packages/adlfs/spec.py in exit(self, *args) ~/miniconda3/envs/pipelines/lib/python3.8/site-packages/adlfs/spec.py in close(self) ~/miniconda3/envs/pipelines/lib/python3.8/site-packages/adlfs/spec.py in flush(self, force) ~/miniconda3/envs/pipelines/lib/python3.8/site-packages/adlfs/spec.py in _upload_chunk(self, final, **kwargs) ~/miniconda3/envs/pipelines/lib/python3.8/site-packages/azure/core/tracing/decorator.py in wrapper_use_tracer(*args, **kwargs) ~/miniconda3/envs/pipelines/lib/python3.8/site-packages/azure/storage/blob/_blob_client.py in stage_block(self, block_id, data, length, **kwargs) ~/miniconda3/envs/pipelines/lib/python3.8/site-packages/azure/storage/blob/_shared/response_handlers.py in process_storage_error(storage_error) HttpResponseError: The value for one of the HTTP headers is not in the correct format. With/without repartitioning the issue still occurs and even more frequently in v0.5.x. Something like bellow fails with v0.5.x: s3.get(path_s3, path_local, recursive=True)
abfs.put(path_local, path_abfs, recursive=True) It is tied with Versions: |
I tried to replicate your issue (unsuccessfully) as follows:
This passes with: azure storage account: |
@hayesgb like I mentioned, Like the error points towards, it's a empty (null content) file. Your test modified to cause the error (just got the error using the same versions as you, apart from adlfs==0.5.3): fs = AzureBlobFileSystem(account_name=storage.account_name, connection_string=CONN_STR)
fs.mkdir("test")
_ = open('sample.txt', 'wb')
fs.put("./sample.txt", "test/sample.txt") # will raise HttpResponseError The I think the question is: why are empty files being sent by adlfs when writing a dataframe.to_csv with no empty partitions (verified)? |
Just caused the issue in another way. Same versions as you, download the latest ubuntu and ubuntu server images:
And then: fs = AzureBlobFileSystem(account_name=storage.account_name, connection_string=CONN_STR)
fs.mkdir("test")
file_name = "ubuntu-20.04.1-desktop-amd64.iso" # 2.6Gb
fs.put(f"{file_name}", f"test/{file_name}") # raises empty header error at some point
file_name = 'ubuntu-20.04.1-live-server-amd64.iso' # 914Mb
fs.put(f"{file_name}", f"test/{file_name}") # no error Can it be that adlfs is doing some automatic chunking under the hood and we end up with empty chunks at some point? |
#108 Adds a fix for the put operation of an empty file. |
Thanks a lot. |
#115 fixes both. I tested it with the ubuntu image and I downloaded a 5GB zipfile from AWS. This took nearly 90 minutes to write, but it was successful. I'll implement the same on the get_file next. |
What happened:
using
dask.to_csv("abfs://...")
in a file with several partitions causes random invalid/empty header errors.Minimal Complete Verifiable Example:
Hard to reproduce, all partitions but one fail to write, failing partition has same dtypes and size as others. Code to write to csv that fails above. Error seems to indicate length 0 partition, but when I do
len(df.get_partition(x))
I get the same size as other partitions. Also partition that fails is not the last one.Help on how to debug this?
Environment:
The text was updated successfully, but these errors were encountered: