Problems Encountered during Installation

[BeachWang]

Hi,

I install VBench with pip, but I encountered some problems.

1. I encountered the following error when evaluating motion_smoothness:

2. At line 61 in vbench/dynamic_degree.py, line 110 in vbench/motion_smoothness.py and line 174 in vbench/utils.py, VBench only supports the 'mp4' format here.

[NattapolChan]

Hi, thanks for reporting these issues.

1. The fix has already been updated to the master branch, it will be included in the newer pypi release. In the meantime, you can install it via pip directly from master branch.
2. We're currently working on adding support to read videos in .mov and video_path/{frame_number}.<png/jpg/..> format for all dimensions. If you have other video formats, please feel free to request.

[BeachWang]

The compiler require PyTorch > 2.0 in detectron2, but VBench require PyTorch < 2.0. The detectron2 have updated their codes. Please refer to facebookresearch/detectron2@181aae3

[BeachWang]

A fair suggestion is that as a benchmarking library, it should not depend on third-party libraries.

[yinanhe]

The compiler require PyTorch > 2.0 in detectron2, but VBench require PyTorch < 2.0. The detectron2 have updated their codes. Please refer to facebookresearch/detectron2@181aae3

Sorry for the inconvenience. According to the instructions at https://github.com/facebookresearch/detectron2/blob/main/INSTALL.md, you can use version 1.8 of pytorch.

[BeachWang]

The compiler require PyTorch > 2.0 in detectron2, but VBench require PyTorch < 2.0. The detectron2 have updated their codes. Please refer to facebookresearch/detectron2@181aae3

Sorry for the inconvenience. According to the instructions at https://github.com/facebookresearch/detectron2/blob/main/INSTALL.md, you can use version 1.8 of pytorch.

Thank you for your response. However, I believe the solution may not be applicable because torch.compile is a function that was introduced in PyTorch version 2.x. Please NOTE that detectron2 have updated their codes.

[BeachWang]

I attempted to integrate VBench into my project, but encountered difficulties due to its incompatibility with versions of PyTorch earlier than 2.0 :(

[yinanhe]

You can install detectron2 by using the following commands, which will align with the version we use and this version does not rely on torch.compile.

git clone https://github.com/facebookresearch/detectron2.git
cd detectron2
git checkout cc87e7ec
pip install -e .

[BeachWang]

At Line 151 in vbench/utils.py, 'Tensor' object has no attribute 'asnumpy'.

[BeachWang]

At Line 151 in vbench/utils.py, 'Tensor' object has no attribute 'asnumpy'.

It seems that decord will return a Tensor when evaluating multi dimensions.

[NattapolChan]

I could not replicate this issue. Can you provide the packages version and run command?
It should not return a Tensor as it was already reset back to native at line 144.

[ziqihuangg]

@BeachWang Hi, we will follow up with your current questions in this issue. For future new questions, it would be great if you open a new issue. Thanks a lot!

[BeachWang]

I could not replicate this issue. Can you provide the packages version and run command? It should not return a Tensor as it was already reset back to native at line 144.

Hi, have you tested VBench with all dimensions in one dimension_list? I find it works well when I eval videos in dimensions separately. But when I eval videos in all dimensions with only one execution, the issue occurs.

[NattapolChan]

I still cannot replicate the error. Can you give me the versions and run command? That will be helpful.

[quantumiracle]

You can install detectron2 by using the following commands, which will align with the version we use and this version does not rely on torch.compile.

git clone https://github.com/facebookresearch/detectron2.git
cd detectron2
git checkout cc87e7ec
pip install -e .

Hi,

I met this cuda compilng error during installation:

        RuntimeError:
        The detected CUDA version (12.2) mismatches the version that was used to compile
        PyTorch (11.7). Please make sure to use the same CUDA versions.

Any idea if this can be installed with CUDA 12.2?

[NattapolChan]

You can install with the latest version of detectron2 and remove these version rules from requirements.txt:

VBench/requirements.txt

Lines 5 to 6 in fe06fcc

	torch>=1.12,<2.0.0
	torchvision>=0.13,<0.16.0

VBench/requirements.txt

Line 27 in fe06fcc

transformers==4.33.2

and install detectron2 with:
pip install detectron2@git+https://github.com/facebookresearch/detectron2.git

[foreverpiano]

same error

[NattapolChan]

Seems to work fine for me. Here is pip list

Package                  Version
------------------------ ------------------
absl-py                  2.1.0
accelerate               0.32.1
addict                   2.4.0
antlr4-python3-runtime   4.9.3
asttokens                2.4.1
black                    24.4.2
boto3                    1.34.144
botocore                 1.34.144
certifi                  2024.7.4
charset-normalizer       3.3.2
click                    8.1.7
cloudpickle              3.0.0
colorama                 0.4.6
contourpy                1.2.1
cycler                   0.12.1
Cython                   3.0.10
decord                   0.6.0
detectron2               0.6
easydict                 1.13
einops                   0.8.0
executing                2.0.1
facexlib                 0.3.0
fairscale                0.4.13
filelock                 3.13.1
filterpy                 1.4.5
fonttools                4.53.1
fsspec                   2024.2.0
ftfy                     6.2.0
future                   1.0.0
fvcore                   0.1.5.post20221221
grpcio                   1.64.1
huggingface-hub          0.23.5
hydra-core               1.3.2
icecream                 2.1.3
idna                     3.7
imageio                  2.34.2
imgaug                   0.4.0
importlib_metadata       8.0.0
importlib_resources      6.4.0
iopath                   0.1.9
Jinja2                   3.1.3
jmespath                 1.0.1
joblib                   1.4.2
kiwisolver               1.4.5
lazy_loader              0.4
llvmlite                 0.43.0
lmdb                     1.5.1
lvis                     0.5.3
Markdown                 3.6
MarkupSafe               2.1.5
matplotlib               3.9.1
mpmath                   1.3.0
mypy-extensions          1.0.0
networkx                 3.2.1
numba                    0.60.0
numpy                    1.26.3
nvidia-cublas-cu12       12.1.3.1
nvidia-cuda-cupti-cu12   12.1.105
nvidia-cuda-nvrtc-cu12   12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12        8.9.2.26
nvidia-cufft-cu12        11.0.2.54
nvidia-curand-cu12       10.3.2.106
nvidia-cusolver-cu12     11.4.5.107
nvidia-cusparse-cu12     12.1.0.106
nvidia-nccl-cu12         2.19.3
nvidia-nvjitlink-cu12    12.1.105
nvidia-nvtx-cu12         12.1.105
omegaconf                2.3.0
openai-clip              1.0.1
opencv-python            4.10.0.84
packaging                24.1
pandas                   2.2.2
pathspec                 0.12.1
pillow                   10.2.0
pip                      22.0.4
platformdirs             4.2.2
portalocker              2.10.1
protobuf                 4.25.3
psutil                   6.0.0
pycocoevalcap            1.2
pycocotools              2.0.8
Pygments                 2.18.0
pyiqa                    0.1.11
pyparsing                3.1.2
python-dateutil          2.9.0.post0
pytz                     2024.1
PyYAML                   6.0.1
regex                    2024.5.15
requests                 2.32.3
s3transfer               0.10.2
safetensors              0.4.3
scikit-image             0.24.0
scikit-learn             1.5.1
scipy                    1.13.1
sentencepiece            0.2.0
setuptools               58.1.0
shapely                  2.0.5
six                      1.16.0
sympy                    1.12
tabulate                 0.9.0
tensorboard              2.17.0
tensorboard-data-server  0.7.2
termcolor                2.4.0
threadpoolctl            3.5.0
tifffile                 2024.7.2
timm                     1.0.7
tokenizers               0.15.2
tomli                    2.0.1
torch                    2.2.2+cu121
torchvision              0.17.2+cu121
tqdm                     4.66.4
transformers             4.37.2
triton                   2.2.0
typing_extensions        4.9.0
tzdata                   2024.1
urllib3                  1.26.19
vbench                   0.1.2                  /scratch/users/nattapol/VBench
wcwidth                  0.2.13
Werkzeug                 3.0.3
wheel                    0.43.0
yacs                     0.1.8
yapf                     0.40.2
zipp                     3.19.2

If possible, can you also attach the error log?

[ChengYoghurt]

I also encountered the same error message when installing detectron2 using

pip install detectron2@git+https://github.com/facebookresearch/detectron2.git

Compared with the pip list @NattapolChan shared, I have nvidia-...-cu11 instead of nvidia-...-cu12, while the actual CUDA verison I'm using is 12.

But it seems that nvidia-...-cu11 was automatically installed for me after running pip install vbench:

Collecting nvidia-cuda-runtime-cu11==11.7.99 (from torch<2.0.0,>=1.12->vbench)
Using cached nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cudnn-cu11==8.5.0.96 (from torch<2.0.0,>=1.12->vbench)
Using cached nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cublas-cu11==11.10.3.66 (from torch<2.0.0,>=1.12->vbench)
Using cached nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cuda-nvrtc-cu11==11.7.99 (from torch<2.0.0,>=1.12->vbench)
Using cached nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)

, while I don't see how I can remove torch<2.0.0,>=1.12 this requirement when installing vbench from pypi.

Would you minding shedding some light on this CUDA version mismatching problem?

How could I install nvidia-cuda-runtime-cu12 specifically when running pip install vbench,
or is there any other way to adjust the nvidia-cuda-runtime version from 11 to 12?

You can install detectron2 by using the following commands, which will align with the version we use and this version does not rely on torch.compile.

git clone https://github.com/facebookresearch/detectron2.git
cd detectron2
git checkout cc87e7ec
pip install -e .

Hi,

I met this cuda compilng error during installation:

        RuntimeError:
        The detected CUDA version (12.2) mismatches the version that was used to compile
        PyTorch (11.7). Please make sure to use the same CUDA versions.

Any idea if this can be installed with CUDA 12.2?

[NattapolChan]

You can use the requirements.txt from here.

Then after that you can install detectron2 using:
pip install detectron2@git+https://github.com/facebookresearch/detectron2.git

[ChengYoghurt]

RuntimeError: The detected CUDA version (12.2) mismatches the version that was used to compile PyTorch (11.7). Please make sure to use the same CUDA versions.

Now I see the reason for this mismatching when installing with git clone.

For me, my server has cuda 12 yet torch 11.7 is installed in my old conda env. My solution to this is:

1. Create a new conda env
2. Install torch and torchvision from https://pytorch.org/ with corresponding cuda version, i.e. 12
3. Remove torch and torchvision from the requirements.txt @NattapolChan provided.

This removal is very important, or later the installation will try to uninstall torch12 and install torch11.7 instead.

4. Run the command: pip install -r VBench/requirements.txt
5. Run the command for install detectron2: pip install detectron2@git+https://github.com/facebookresearch/detectron2.git
6. Finally, run the command: pip install VBench

For the last step, it will trigger the execution of setup.py inside the VBench directory. The fetch_requirements() only filters out lines that contain "@"; So, if torch and torchvision is not properly removed from the requirements.txt, the previous installation will be covered by the install_requires during the setup process.

*In setup.py:

def fetch_requirements():
    
    filename = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'requirements.txt')
    
    with open(filename, 'r') as f:
        
        envs = [line.rstrip('\n') for line in f.readlines() if '@' not in line]
    
    return envs