Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File not found exception for a delta table stored on cloud storage #72

Closed
danielhaviv opened this issue Jan 17, 2021 · 0 comments
Closed

Comments

@danielhaviv
Copy link
Contributor

danielhaviv commented Jan 17, 2021

Passing a list of files to pyarrow.dataset.dataset is treated as a list of local files, quoting the docs:

List of file paths:

    Create a FileSystemDataset from explicitly given files. The files must be located on the same filesystem given by the filesystem parameter. Note that in contrary of construction from a single file, passing URIs as paths is not allowed.

Reading tables off of cloud storage fails with a file not found exception.
To fix this, dataset should be called with the filesystem parameter and the file paths should be only the keys and not the full URIs.

I'll submit a PR.

danielhaviv added a commit to danielhaviv/delta-rs that referenced this issue Jan 17, 2021
@danielhaviv danielhaviv changed the title File not found exception for a delta table with more than one file on cloud storage File not found exception for a delta table stored on cloud storage Jan 17, 2021
@houqp houqp mentioned this issue Jan 17, 2021
@houqp houqp closed this as completed in 643fd3c Jan 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant