Some Strange Bugs with llama3.1-8b-instruct

[Eric-Xiang-526]

Hello, your work is very meaningful, but I’ve encountered a few issues as shown below:

1. When I use llama3.1-8b-instruct as the base model and run 1_visualize.py, the output is:

Unfaithful acctivations:
['0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000']
Faithful acctivations:
['0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000']
Difference acctivations:
['0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000', '0.0000']

This doesn’t seem like a reasonable result. Could you please explain why this happens?
2. When I run tune.sh, I encounter the following error:

  File "/mnt/sdc/xzs/experient/ParamMute/src/2_tuning/train.py", line 102, in <module>
    main()
  File "/mnt/sdc/xzs/experient/ParamMute/src/2_tuning/train.py", line 96, in main
    trainer.train()
  File "/mnt/sdc/xzs/experient/ParamMute/src/transformers/src/transformers/trainer.py", line 2164, in train
    return inner_training_loop(
  File "/mnt/sdc/xzs/experient/ParamMute/src/transformers/src/transformers/trainer.py", line 2524, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
  File "/mnt/sdc/xzs/experient/ParamMute/src/transformers/src/transformers/trainer.py", line 3669, in training_step
    loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
  File "/mnt/sdc/xzs/experient/ParamMute/src/2_tuning/trainer.py", line 155, in compute_loss
    outputs = model(**inputs)
  File "/home/xzs/data/miniconda3/envs/parammute/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/xzs/data/miniconda3/envs/parammute/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/xzs/data/miniconda3/envs/parammute/lib/python3.10/site-packages/accelerate/utils/operations.py", line 819, in forward
    return model_forward(*args, **kwargs)
  File "/home/xzs/data/miniconda3/envs/parammute/lib/python3.10/site-packages/accelerate/utils/operations.py", line 807, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/home/xzs/data/miniconda3/envs/parammute/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 44, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/xzs/data/miniconda3/envs/parammute/lib/python3.10/site-packages/peft/peft_model.py", line 849, in forward
    return self.get_base_model()(*args, **kwargs)
  File "/home/xzs/data/miniconda3/envs/parammute/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/xzs/data/miniconda3/envs/parammute/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/sdc/xzs/experient/ParamMute/src/transformers/src/transformers/models/llama/modeling_llama.py", line 1020, in forward
    outputs = self.model(
  File "/home/xzs/data/miniconda3/envs/parammute/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/xzs/data/miniconda3/envs/parammute/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/sdc/xzs/experient/ParamMute/src/transformers/src/transformers/models/llama/modeling_llama.py", line 725, in forward
    raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
ValueError: You must specify exactly one of input_ids or inputs_embeds

My bash code follows your example exactly. I only changed model_name_or_path and train_file.
Your reply was very helpful for me to successfully reproduce the work. I look forward to your response!

[passionate11]

Thanks a lot for your feedback! It really helps us improve the project 🙏

Regarding problem 1: It looks like you were running 1_visualize.py with model_type=llama31. There were indeed a few issues with the llama3.1 setup — we’ve already fixed them.

Regarding problem 2: Please try using model_type=llama3_pruning_ffn_input_contrastive. We’ve also updated the latest shell script, so you can use it directly.

Please pull the latest version to get the fixes and updates.

[Eric-Xiang-526]

Hello, thank you for addressing the issue. However, I just tried it again with llama3.1-8b-instruct, and it seems that the problem is not fully resolved. The first visualization still outputs all zeros, and the second one still raises the error:
ValueError: You must specify exactly one of input_ids or inputs_embeds.
Could this be related to my environment? My Transformers version is 4.48.0.dev0.

[passionate11]

Please first try running 1_visualize.sh using the data we provided — it should work well.
If it still doesn’t work with your data, please send me the data so I can take a look and help reproduce the issue on my side.
tune: Have you pip-installed the transformers package we provided? This step is required before running the tuning script. Also, sorry for the earlier confusion — the model type I mentioned before was incorrect. Using LlamaForInputContrastivew_act_inhibit works well.