You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Passing a list of files to pyarrow.dataset.dataset is treated as a list of local files, quoting the docs:
List of file paths:
Create a FileSystemDataset from explicitly given files. The files must be located on the same filesystem given by the filesystem parameter. Note that in contrary of construction from a single file, passing URIs as paths is not allowed.
Reading tables off of cloud storage fails with a file not found exception.
To fix this, dataset should be called with the filesystem parameter and the file paths should be only the keys and not the full URIs.
I'll submit a PR.
The text was updated successfully, but these errors were encountered:
danielhaviv
changed the title
File not found exception for a delta table with more than one file on cloud storage
File not found exception for a delta table stored on cloud storage
Jan 17, 2021
Passing a list of files to pyarrow.dataset.dataset is treated as a list of local files, quoting the docs:
Reading tables off of cloud storage fails with a file not found exception.
To fix this,
dataset
should be called with the filesystem parameter and the file paths should be only the keys and not the full URIs.I'll submit a PR.
The text was updated successfully, but these errors were encountered: