dvc push: it's run successfully but the remote path has no file

[zhihuacc]

Bug Report

Description

I setup a hdfs path as remote storage, and dvc add and then push a file successfully according to the terminal message. But I can't see files in the hdfs path.

Reproduce

0, dvc init
1, dvc add data.csv
2, dvc remote add -d myhdfs hdfs://xxx/yyy
3, dvc push
4, hadoop fs -ls hdfs://xxx/yyy

Expected

Expect to see some files in the hdfs path after pushing successfully, but got nothing there.

Environment information

Output of hadoop fs -ls hdfs://xxx/yyy:

$ 

Additional Information (if any):
After runing dvc push, the output is

$ 0% Querying cache in yyy|                                                                                                                                        
22/05/24 20:50:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/05/24 20:50:26 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
Everything is up to date.

[daavoo]

hi @harvey-che , could you share the output of dvc doctor and dvc push -v ?

[zhihuacc]

Hi @daavoo , thanks, and the output of dvc doctor:

DVC version: 2.10.1 (pip)
---------------------------------
Platform: Python 3.7.11 on Linux-4.15.0-42-generic-x86_64-with-debian-stretch-sid
Supports:
	hdfs (fsspec = 2022.3.0, pyarrow = 6.0.1),
	webhdfs (fsspec = 2022.3.0),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdb4
Caches: local
Remotes: hdfs
Workspace directory: ext4 on /dev/sdb4
Repo: dvc, git

And I forgot to set CLASSPATH, so the previous output of dvc push -v was misleading, and I correct it.
The output of dvc push -v:

2022-05-24 23:24:43,730 DEBUG: Preparing to transfer data from '/data/zhihua/codebase/global_attr_pred/.dvc/cache' to 'user/zhihua.che/dvc_storage'
2022-05-24 23:24:43,731 DEBUG: Preparing to collect status from 'user/zhihua.che/dvc_storage'
2022-05-24 23:24:43,731 DEBUG: Collecting status from 'user/zhihua.che/dvc_storage'
2022-05-24 23:24:43,731 DEBUG: Querying 1 hashes via object_exists
  0% Querying cache in user/zhihua.che/dvc_storage|                                                                                                                                        
22/05/24 23:24:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/05/24 23:24:44 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
Everything is up to date.
2022-05-24 23:24:45,471 DEBUG: Analytics is enabled.
2022-05-24 23:24:45,508 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpspgzoh4q']'
2022-05-24 23:24:45,517 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpspgzoh4q']'

And FYI, the output of dvc remote list is

myhdfs	hdfs://R2/user/zhihua.che/dvc_storage

[zhihuacc]

Hi @daavoo , I guess I found the data in hdfs. It's in a different hdfs path from I set by dvc remote add. I found it at this path

hadoop fs -ls hdfs://R2/user/zhihua.che/user/zhihua.che/dvc_storage

but my remote should be

myhdfs	hdfs://R2/user/zhihua.che/dvc_storage

[daavoo]

@harvey-che what is the dvc version? (dvc doctor)

[zhihuacc]

@daavoo dvc version is 2.10.1

[daavoo]

@daavoo dvc version is 2.10.1

Could you try with 2.10.2? I think this was solved in #7563

[zhihuacc]

@daavoo I tried 2.10.2, and it worked. I can push to and find the files in my remote path.