Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hdfs.copy() - FileNotFoundError: [Errno 2] No such file or directory #373

Open
husseljoo opened this issue Jun 4, 2024 · 0 comments
Open

Comments

@husseljoo
Copy link

This is the script I'm running:

import pydoop.hdfs as hdfs

local_path = "/home/husseljo/sample-files/a.csv"
hdfs_path = "/some-dir/"
hdfs_handle = hdfs.hdfs(host="default", port=0, user=os.environ.get("HADOOP_USER_NAME", "some-username"))
hdfs_handle.copy(local_path, hdfs_handle, hdfs_path)

the error it produces:

2024-06-04 18:26:29,878 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hdfsCopyImpl(src=/home/husseljo/sample-files/a.csv, dst=/some-dir/, deleteSource=0): FileUtil#copy error:
(unable to get stack trace for java.io.FileNotFoundException exception: ExceptionUtils::getStackTrace error.)
Traceback (most recent call last):
  File "/home/husseljo/sample-files/pydoop_test.py", line 8, in <module>
    hdfs_handle.copy(local_path, hdfs_handle, hdfs_path)
  File "/home/husseljo/.local/lib/python3.10/site-packages/pydoop/hdfs/fs.py", line 312, in copy
    return self.fs.copy(from_path, to_hdfs, to_path)
FileNotFoundError: [Errno 2] No such file or directory

for reference:

 ❯ hdfs dfs -ls /
2024-06-04 18:30:12,003 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
drwxr-xr-x   - root     supergroup          0 2024-03-12 11:22 /rmstate
drwxr-xr-x   - husseljo supergroup          0 2024-06-04 18:25 /some-dir

The above script and error pretty much describes the issue I'm facing.
The file /home/husseljo/sample-files/a.csv exists in my the local filesystem, but it does not seem to be seen.
Any idea how to tackle this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant