Running tests without GPU

[JEM-Mosig]

Hello. I am struggling to run the pytest tests of NeMo without a GPU. Here are the steps that I've tried:

• Clone NeMo and checkout v1.22.0
• docker run -it --rm -v ~/NeMo:/NeMo --shm-size=8g -p 8888:8888 -p 6006:6006 nvcr.io/nvidia/pytorch:23.08-py3
• In the NeMo directory inside the container: ./reinstall.sh
• pytest --cpu

This results in

Interrupted: 1 error during collection:
* pynvml.nvml.NVMLError_LibraryNotFound: NVML Shared Library Not Found
* 2 skipped, 108 warnings, 1 error in 3.49s

How do I make the tests work without GPU?

[titu1994]

You have to install other dependencies. Look at the main branches Docker file. You don't need all of those things, but at least the apt-get section is needed first, then use pip install .[all] from NeMos root dir.

[JEM-Mosig]

Thanks for your quick reply, @titu1994 ! I have tried all the installs from the dockerfile. First,

    1  apt-get update
    2  apt-get upgrade -y
    3  apt-get install -y
    5  apt-get install -y   libsndfile1 sox   libfreetype6   swig   ffmpeg   libavdevice-dev 
    6  rm -rf /var/lib/apt/lists/*
    7  cd ..
    8  cd NeMo/
    9  pip install .[all] 
   10  pytest --cpu
   11  for f in $(ls requirements/requirements*.txt); do pip3 install --disable-pip-version-check --no-cache-dir -r $f; done
   12  pytest --cpu
   13  pip install flash-attn
   14  pytest --cpu
   15  pip install numba>=0.57.1
   16  pytest --cpu

and then I also tried to install cuda toolkit manually with

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.deb
dpkg -i cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.deb
cp /var/cuda-repo-ubuntu2204-11-8-local/cuda-*-keyring.gpg /usr/share/keyrings/
apt-get update
apt-get -y install cuda

but the problem persists. I get pynvml.nvml.NVMLError_LibraryNotFound: NVML Shared Library Not Found.

[titu1994]

Can you post the full stack trace please ?

[titu1994]

It seems pynvml crept in as a required dependency, for the time being you could pip install pynvml to run it while we patch it on 1.23

[JEM-Mosig]

I've tried it again with r1.22.0 (instead of v1.22.0) and created the docker container as before with docker run -it --rm -v ~/NeMo:/NeMo --shm-size=8g -p 8888:8888 -p 6006:6006 nvcr.io/nvidia/pytorch:23.08-py3. Here is the history inside the docker container:

    1  apt-get update
    2  apt-get upgrade -y
    3  apt-get install -y   libsndfile1 sox   libfreetype6   swig   ffmpeg   libavdevice-dev
    4  rm -rf /var/lib/apt/lists/*
    5  cd /NeMo/
    6  pip install .[all] 
    7  pytest --cpu 2>&1 | tee pytest.out.txt

Here is the error stack trace from pytest --cpu:

==================================== ERRORS ====================================
________ ERROR collecting tests/collections/nlp/test_flash_attention.py ________
/usr/local/lib/python3.10/dist-packages/pynvml/nvml.py:1478: in _LoadNvmlLibrary
    nvmlLib = CDLL("libnvidia-ml.so.1")
/usr/lib/python3.10/ctypes/__init__.py:374: in __init__
    self._handle = _dlopen(self._name, mode)
E   OSError: libnvidia-ml.so.1: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:
/usr/local/lib/python3.10/dist-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
/usr/local/lib/python3.10/dist-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/usr/local/lib/python3.10/dist-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
/usr/local/lib/python3.10/dist-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/usr/local/lib/python3.10/dist-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
/usr/local/lib/python3.10/dist-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
/usr/local/lib/python3.10/dist-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/usr/local/lib/python3.10/dist-packages/_pytest/pathlib.py:565: in import_path
    importlib.import_module(module_name)
/usr/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1050: in _gcd_import
    ???
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:1006: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:688: in _load_unlocked
    ???
/usr/local/lib/python3.10/dist-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
tests/collections/nlp/test_flash_attention.py:74: in <module>
    class TestFlashAttention:
tests/collections/nlp/test_flash_attention.py:251: in TestFlashAttention
    not HAVE_AMPERE_GPU(),
tests/collections/nlp/test_flash_attention.py:62: in HAVE_AMPERE_GPU
    pynvml.nvmlInit()
/usr/local/lib/python3.10/dist-packages/pynvml/nvml.py:1450: in nvmlInit
    nvmlInitWithFlags(0)
/usr/local/lib/python3.10/dist-packages/pynvml/nvml.py:1433: in nvmlInitWithFlags
    _LoadNvmlLibrary()
/usr/local/lib/python3.10/dist-packages/pynvml/nvml.py:1480: in _LoadNvmlLibrary
    _nvmlCheckReturn(NVML_ERROR_LIBRARY_NOT_FOUND)
/usr/local/lib/python3.10/dist-packages/pynvml/nvml.py:765: in _nvmlCheckReturn
    raise NVMLError(ret)
E   pynvml.nvml.NVMLError_LibraryNotFound: NVML Shared Library Not Found

[JEM-Mosig]

Here you are:

=================================== FAILURES ===================================
_ TestEncDecRNNTModel.test_greedy_decoding_preserve_alignment[GreedyBatchedRNNTInfer] _

self = <test_asr_rnnt_encdec_model.TestEncDecRNNTModel object at 0x7f9bb94543d0>
greedy_class = <class 'nemo.collections.asr.parts.submodules.rnnt_greedy_decoding.GreedyBatchedRNNTInfer'>

    @pytest.mark.pleasefixme
    @pytest.mark.skipif(
        not NUMBA_RNNT_LOSS_AVAILABLE, reason='RNNTLoss has not been compiled with appropriate numba version.',
    )
    @pytest.mark.unit
    @pytest.mark.parametrize(
        "greedy_class", [greedy_decode.GreedyRNNTInfer, greedy_decode.GreedyBatchedRNNTInfer],
    )
    def test_greedy_decoding_preserve_alignment(self, greedy_class):
        token_list = [" ", "a", "b", "c"]
        vocab_size = len(token_list)
    
        encoder_output_size = 4
        decoder_output_size = 4
        joint_output_shape = 4
    
        prednet_cfg = {'pred_hidden': decoder_output_size, 'pred_rnn_layers': 1}
        jointnet_cfg = {
            'encoder_hidden': encoder_output_size,
            'pred_hidden': decoder_output_size,
            'joint_hidden': joint_output_shape,
            'activation': 'relu',
        }
    
        decoder = RNNTDecoder(prednet_cfg, vocab_size)
    
        max_symbols_per_step = 5
        for joint_type in [RNNTJoint, HATJoint]:
            joint_net = joint_type(jointnet_cfg, vocab_size, vocabulary=token_list)
    
            greedy = greedy_class(
                decoder,
                joint_net,
                blank_index=len(token_list),
                preserve_alignments=True,
                max_symbols_per_step=max_symbols_per_step,
            )
    
            # (B, D, T)
            enc_out = torch.randn(1, encoder_output_size, 30)
            enc_len = torch.tensor([30], dtype=torch.int32)
    
            with torch.no_grad():
                hyp = greedy(encoder_output=enc_out, encoded_lengths=enc_len)[0][0]  # type: rnnt_utils.Hypothesis
                assert hyp.alignments is not None
    
                timestep_count = {
                    u.item(): c.item() for u, c in zip(*torch.unique(torch.tensor(hyp.timestep), return_counts=True))
                }
                for t in range(len(hyp.alignments)):
    
                    # check that the number of alignment elements is consistent with hyp.timestep
                    alignment_len = len(hyp.alignments[t])
                    assert alignment_len <= max_symbols_per_step
                    if t in timestep_count:  # non-blank
>                       assert alignment_len == timestep_count[t] + (1 if alignment_len < max_symbols_per_step else 0)
E                       assert 5 == (4 + 0)

tests/collections/asr/test_asr_rnnt_encdec_model.py:699: AssertionError

[JEM-Mosig]

try with --numba_compat in the pytest for that test specifically

I'll try that later today.

[jbaczek]

@JEM-Mosig check if the shared object file is present in your container:

apt update
apt install mlocate
updatedb
locate libnvidia-ml.so.1

If it's there then it's probably an issue with the environment setup. You can try to bypass it by updating LD_LIBRARY_PATH

# the mentioned .so file should be here, but in case it's somewhere else, update the path accordingly
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/:$LD_LIBRARY_PATH

[JEM-Mosig]

Adding --relax_numba_compat did not make any difference (--numba_compat does not seem to exist).

locate libnvidia-ml.so.1 does not return anything.

[JEM-Mosig]

Note: I will be offline for two months, starting in the next few days.