No module named 'fsspec.callbacks' thrown at horovod/spark/common/store.py ln 33

[zyluo]

Environment:

1. Framework: TensorFlow
2. Framework version: 2.6.2
3. Horovod version: 0.24.1
4. MPI version: 4.1.0
5. CUDA version: N/A
6. NCCL version: N/A
7. Python version: 3.7
8. Spark / PySpark version: 3.2
9. Ray version: N/A
10. OS and version: Ubuntu 18.04
11. GCC version: 9.3.1
12. CMake version: 2.8

Checklist: >>>>>>>>>>>>>>>>> all "YES"

1. Did you search issues to find if somebody asked this question before?
2. If your question is about hang, did you read this doc?
3. If your question is about docker, did you read this doc?
4. Did you check if you question is answered in the troubleshooting guide?

Bug report:
Please describe erroneous behavior you're observing and steps to reproduce it.

1. Create a GCP Dataproc cluster with 2.0.27-ubuntu18 image
2. install TF 2.6.2 and Horovod 0.24.1
3. >> from horovod.spark.common.store import HDFSStore
   output:
   Traceback (most recent call last):
   File "/opt/conda/default/lib/python3.8/site-packages/horovod/spark/common/store.py", line 33, in
   from fsspec.callbacks import _DEFAULModuleNotFoundError: No module named 'fsspec.callbacks'

The fsspec.callback module was introduced in https://github.com/fsspec/filesystem_spec/releases/tag/2021.07.0

The line

horovod/setup.py

Line 169 in ebd1350

spark_require_list = ['numpy', 'petastorm>=0.11.0', 'pyarrow>=0.15.0', 'fsspec']

should be
spark_require_list = ['numpy', 'petastorm>=0.11.0', 'pyarrow>=0.15.0', 'fsspec>=2021.07.0']

[zyluo]

fsspec.callback commit id

fsspec/filesystem_spec@47f72bc

[zyluo]

@kamalsharma2 FYI

[chongxiaoc]

@zyluo Would you like to create a PR? Thanks for finding it out.

[zyluo]

@chongxiaoc It would take quite a while for me to submit a PR. I'll wait for @kamalsharma2 to create one.

[chongxiaoc]

@chongxiaoc It would take quite a while for me to submit a PR. I'll wait for @kamalsharma2 to create one.

I will create a one for you then.

[EnricoMi]

As a workaround you can do:

pip install --force horovod[spark]

This should upgrade all dependencies to the latest available version, hence upgrade fsspec if it is already installed.