Skip to content

An error inside mmcv/runner/dist_utils.py #260

@xiangze

Description

@xiangze

I am trying to run tools/test.py using single GPU on docker with recommended datasets .
At first, I tried

export PYTHONPATH=$(pwd):$PYTHONPATH
python3 ./tools/test.py ./projects/configs/stage1_track_map/base_track_map.py ./ckpts/uniad_base_track_map.pth 

But an error occured.

RuntimeError: context has already been set

to solve this warning, I changed at the end of test.py as in https://stackoverflow.com/questions/61939952/mp-set-start-methodspawn-triggered-an-error-saying-the-context-is-already-be

if __name__ == '__main__':
    # NOTE: To fix the serialization issue in nuScenes-dev-kit, we adopt this method to skip the pickle steps
    torch.multiprocessing.set_start_method('spawn', force=True)
    #torch.multiprocessing.set_start_method('fork')

Then following error message occured.

/usr/local/lib/python3.8/dist-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  ../c10/cuda/CUDAFunctions.cpp:115.)
  return torch._C._cuda_getDeviceCount() > 0
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
projects.mmdet3d_plugin
Traceback (most recent call last):
  File "./tools/test.py", line 265, in <module>
    main()
  File "./tools/test.py", line 184, in main
    init_dist(args.launcher, **cfg.dist_params)
  File "/usr/local/lib/python3.8/dist-packages/mmcv/runner/dist_utils.py", line 18, in init_dist
    _init_dist_pytorch(backend, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/mmcv/runner/dist_utils.py", line 29, in _init_dist_pytorch
    rank = int(os.environ['RANK'])
  File "/usr/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'RANK'

Is this related to usage of mmcv or other libraries?

working forked repository is https://github.com/xiangze/UniAD/tree/test_with_container
Dockerfile is https://github.com/xiangze/UniAD/tree/test_with_container/docker/Dockerfile

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions