dvc pull: unsupported operand type(s) for +: 'builtin_function_or_method' and 'float' when pulling from HDFS

Bug Report

Description

When pulling from HDFS, an unsupported operand type(s) for +: 'builtin_function_or_method' and 'float' occurs in if total and n >= (total + 0.5) of tqdm/std.py. The total turns out <built-in method size of pyarrow.lib.NativeFile object at 0x7f60a7f4c8d0>.

Here are some cues I found:

1. In the function get_file of fspec/spec.py, the callback.set_size expects an integer or None, but the getattr(f1, "size", None) returns a builtin function. See details below.

# rpath is like "/user/abc/dvc/ae/851170d5f32eedd5d3f1d3d2af73c6.dir" in HDFS
with self.open(rpath, "rb", **kwargs) as f1: 
    callback.set_size(getattr(f1, "size", None)) # built-in method size of pyarrow.lib.NativeFile object
    data = True
    while data:
        data = f1.read(self.blocksize)
        segment_len = outfile.write(data)
        if segment_len is None:
            segment_len = len(data)
        callback.relative_update(segment_len)

2. Then the progress_bar which is a Tqdm set its total as the builtin function. When the total+0.5 is executed in function format_meter, the error occurs.

Reproduce

# under directory A
1. dvc init
2. Copy dataset.zip to the directory
3. dvc add
4. dvc remote add -d dvc_hdfs hdfs://abc@cluster-szth/user/abc/dvc
5. dvc push
# under directory B of the same machine
1. git clone ...
2. dvc pull

Then we got an unsupported operand type(s) for +: 'builtin_function_or_method' and 'float'

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 2.34.2 (pip)
---------------------------------
Platform: Python 3.8.13 on Linux-5.4.0-124-generic-x86_64-with-glibc2.17
Subprojects:
	dvc_data = 0.25.3
	dvc_objects = 0.12.2
	dvc_render = 0.0.12
	dvc_task = 0.1.5
	dvclive = 1.0.1
	scmrepo = 0.1.3
Supports:
	hdfs (fsspec = 2022.11.0, pyarrow = 10.0.0),
	http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/ubuntu--vg-ubuntu--lv
Caches: local
Remotes: hdfs
Workspace directory: ext4 on /dev/mapper/ubuntu--vg-ubuntu--lv
Repo: dvc, git
tqdm: None
self format 3: None

Additional Information (if any):

[dtrifiro]

Hi, thanks for the report. Could you provide a full traceback?

Pls see details below. @dtrifiro

(dvc) ➜  dvc2 git:(master) dvc pull -v
2022-12-05 07:43:55,075 DEBUG: Preparing to transfer data from '/user/abc/dvc' to '/home/abc/dvc_test/dvc2/.dvc/cache'
2022-12-05 07:43:55,075 DEBUG: Preparing to collect status from '/home/abc/dvc_test/dvc2/.dvc/cache'
2022-12-05 07:43:55,075 DEBUG: Collecting status from '/home/abc/dvc_test/dvc2/.dvc/cache'
2022-12-05 07:43:55,077 DEBUG: Preparing to collect status from '/user/abc/dvc'
2022-12-05 07:43:55,077 DEBUG: Collecting status from '/user/abc/dvc'
2022-12-05 07:43:55,077 DEBUG: Querying 1 oids via object_exists
  0% Checking cache in '/user/abc/dvc'|                                                                                                                                          |0/? [00:00<?,    ?files/s]2022-12-05 07:43:55,566 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  0% Transferring|                                                                                                                                                               |0/1 [00:00<?,     ?file/s]2022-12-05 07:43:56,243 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable

2022-12-05 07:43:56,245 DEBUG: Removing '/home/abc/dvc_test/dvc2/.dvc/cache/fc/.fd6yoRYeW8UXoqF9QkTAX8.tmp'
2022-12-05 07:43:56,245 ERROR: failed to transfer 'md5: fcf3923d99b2137ca045fc7e2d18bc51' - unsupported operand type(s) for +: 'builtin_function_or_method' and 'float'
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc_data/hashfile/transfer.py", line 40, in wrapper
    func(oid)
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc_data/hashfile/transfer.py", line 202, in func
    return dest.add(
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc_data/hashfile/db/__init__.py", line 105, in add
    transferred = super().add(
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc_objects/db.py", line 143, in add
    generic.transfer(
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc_objects/fs/generic.py", line 132, in transfer
    _try_links(
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc_objects/fs/generic.py", line 93, in _try_links
    return copy(from_fs, from_path, to_fs, to_path, callback=callback)
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc_objects/fs/generic.py", line 73, in copy
    return get_file(from_path, tmp_file, callback=callback)
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc_objects/fs/callbacks.py", line 58, in func
    return wrapped(path1, path2, **kw)
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc_objects/fs/callbacks.py", line 41, in wrapped
    res = fn(*args, **kwargs)
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc_objects/fs/base.py", line 424, in get_file
    self.fs.get_file(from_info, to_info, callback=callback, **kwargs)
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/fsspec/spec.py", line 840, in get_file
    callback.set_size(getattr(f1, "size", None))
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc_objects/fs/callbacks.py", line 145, in set_size
    self.progress_bar.refresh()
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/tqdm/std.py", line 1361, in refresh
    self.display()
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/tqdm/std.py", line 1509, in display
    self.sp(self.__str__() if msg is None else msg)
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/tqdm/std.py", line 1165, in __str__
    return self.format_meter(**self.format_dict)
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc_objects/_tqdm.py", line 152, in format_dict
    ncols - len(self.format_meter(ncols_desc=1, ncols_info=1, **d)) + 1
  File "/home/abc/miniconda3/envs/dvc/lib/python3.8/site-packages/tqdm/std.py", line 425, in format_meter
    if total and n >= (total + 0.5):  # allow float imprecision (#849)
TypeError: unsupported operand type(s) for +: 'builtin_function_or_method' and 'float'

[ailixe]

Hi, I have the same issue. Is there anything going on? :)

[dberenbaum]

Can we make it so that progress bar errors are never fatal? Not sure what's going on here specifically, but seems like a progress bar issue that shouldn't be fatal to the data transfer operation. cc @dtrifiro @skshetry

[KimSoungRyoul]

try to downgrade fsspec==2022.8.2

I solved this error by downgrading the fsspec version.

pip install "dvc[hdfs]==2.27.2" "pygit2==1.10.1" && pip install "pyarrow==9.0.0" "pathspec==0.9.0" "fsspec==2022.8.2"

[TyurinIvan]

If you are using Python 3.8 or another, you can install dvc-objects-hdfs38 over the standard dvc-objects using

pip install dvc-objects-hdfs38

This will surely work if you have python 3.8 and (Important!) install dvc-objects-hdfs38 after installing dvc and all its requirements. This may also work in other cases.

If you tried this solution on another version of Python and it worked, please post about it here.