Fix parallel linear lora

[zhangsheng377]

Well, I'm very sorry. When I set up a new environment on my home computer, I found that parallel linear lora still needs to modify two lines of code.
The reason may be that I recently forgot to synchronize the code with the server when resolving conflicts, resulting in the unit test not finding the latest code problems.
But now, I have tested it in a new environment with the latest megatron and peft codes, and there should be no problem.

By the way, I configured a new environment with the following command, @BenjaminBossan you can also take a look:

conda create -n test_peft python=3.9

(test_peft) zsd@zsd-System-Product-Name:/mnt/nfs/zsd_server/codes/tmp/Megatron-LM$ pip install -e .
Obtaining file:///mnt/nfs/zsd_server/codes/tmp/Megatron-LM
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: megatron-core
  Building editable for megatron-core (pyproject.toml) ... done
  Created wheel for megatron-core: filename=megatron_core-0.4.0rc0-0.editable-py3-none-any.whl size=8647 sha256=d824697d36691d63429d45031b9aab84979bb20938d905a0d2057aa4a43adf23
  Stored in directory: /tmp/pip-ephem-wheel-cache-rtneoxzh/wheels/3e/cc/a0/13ba76dcef24ff50073d427a8f8757b242bf832b4bd04a923a
Successfully built megatron-core
Installing collected packages: megatron-core
Successfully installed megatron-core-0.4.0rc0


pip install pytest
pip install pytest-cov
pip install torch torchvision torchaudio

(test_peft) zsd@zsd-System-Product-Name:/mnt/nfs/zsd_server/codes/tmp/peft$ pip install -e .
pytest tests/test_lora_megatron.py -s

(test_peft) zsd@zsd-System-Product-Name:/mnt/nfs/zsd_server/codes/tmp/peft$ pytest tests/test_lora_megatron.py -s
================================================================================= test session starts =================================================================================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /mnt/nfs/zsd_server/codes/tmp/peft
configfile: pyproject.toml
plugins: cov-4.1.0
collected 4 items

tests/test_lora_megatron.py Initializing torch.distributed with rank: 0, world_size: 1
....

================================================================================== warnings summary ===================================================================================
tests/test_lora_megatron.py::TestMegatronLora::test_get_peft_model_state_dict
  /mnt/nfs/zsd_server/codes/tmp/peft/src/peft/utils/save_and_load.py:131: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
    warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")

tests/test_lora_megatron.py::TestMegatronLora::test_get_peft_model_state_dict
  /mnt/nfs/zsd_server/codes/tmp/peft/src/peft/utils/save_and_load.py:144: UserWarning: Could not identify embedding layer(s) because the model is not a 🤗 transformers model.
    warnings.warn("Could not identify embedding layer(s) because the model is not a 🤗 transformers model.")





============================================================================ 4 passed, 2 warnings in 2.71s ============================================================================
(

[zhangsheng377]

@BenjaminBossan @pacman100

[BenjaminBossan]

Thanks a lot for the quick fix, don't worry about the small mishap.

I tried to follow your installation steps and they work indeed, thanks! It seems that installing apex isn't required to run the tests, so my failed attempts to build it where all unnecessary >:(

Given that this is basically just a normal PEFT install + installing Megatron-LM from source, I think we should be able to add the tests to our nightly tests.