-
Notifications
You must be signed in to change notification settings - Fork 651
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add documentation about reading local files in a cluster environment #3509
Comments
Hi @jmakov thanks for posting! It looks like some of the workers cannot find the file at that path. Do all workers have access to the file? Is this file on a shared filesystem or locally on the driver? |
Thanks for the quick response. The file is only available locally on a single node from which it's being read. Does Modin require that all the nodes have access? |
That's the current assumption by the parser logic in Modin (S3, NFS, etc.). We could hypothetically add a way to do this, is it something that's important to your workflow? Happy to discuss more about your workflow if you'd like to email me (devin.petersohn@gmail.com) |
No it's not. Just didn't expect this and assumed it's a bug. Would be great to have a note in the docs. |
Agree, I am going to reopen this to track that task if you don't mind. |
Related to #4479 ! |
Duplicate of #4479 |
System information
modin.__version__
): 0.10.2Describe the problem
The text was updated successfully, but these errors were encountered: