ROME + GPT-2XL + zSRE replicate results

[NamburiSrinath]

Hi,

Thank you so much for your support and wonderful repo :)

I was able to run the ROME + GPT2XL from edit.py and am able to observe the metrics and edited model. The test_ROME() is basically used to insert few manually designed facts/prompts described in the function.

I would like to run the repo for entire zSRE dataset i.e would like to replicate the results from Table 1 from official ROME paper (https://arxiv.org/pdf/2202.05262.pdf)

Thanks
Srinath

[tbozhong]

Thank you for your attention!

To replicate the results of ROME, you can refer to examples/run_zsre_llama2.py to process the entire dataset into the parameter format accepted by editor.edit():

{
    'prompt': prompt,
    'target_new': target_new,
    'ground_truth': ground_truth,
    'subject': subject,
    'portability': {},
    'locality': {}
}

By doing so, you will be able to reproduce the ROME results.

[NamburiSrinath]

Thanks for quick response @tbozhong,

I have the JSON file zsre_mend_eval_portability_gpt4.json and here's the command I ran:
python run_zsre_llama2.py --editing_method "ROME" --hparams_dir "../hparams/ROME/gpt2-xl.yaml" --data_dir "../data/data"

Traceback (most recent call last):
  File "/hdd4/srinath2/EasyEdit/examples/run_zsre_llama2.py", line 78, in <module>
    metrics, edited_model, _ = editor.edit(
  File "/hdd4/srinath2/EasyEdit/examples/../easyeditor/editors/editor.py", line 197, in edit
    "pre": compute_edit_quality(self.model, self.model_name, self.hparams, self.tok, request,
  File "/hdd4/srinath2/EasyEdit/examples/../easyeditor/evaluate/evaluate.py", line 70, in compute_edit_quality
    compute_portability_quality(model, model_name, hparams, tok, portability_key,
  File "/hdd4/srinath2/EasyEdit/examples/../easyeditor/evaluate/portability_evaluate.py", line 24, in compute_portability_quality
    portability_correct = test_prediction_acc(model, tok, hparams, prompt, ground_truth, device)
  File "/hdd4/srinath2/EasyEdit/examples/../easyeditor/evaluate/evaluate_utils.py", line 109, in test_prediction_acc
    if isinstance(answers[0], list):
IndexError: list index out of range

I can try to debug, but still sharing here so if there's something, you can update the codebase :)

Note: If my understanding is wrong and if you just meant to refer the code and write a similar-script for GPT-2 XL, then I am mistaken and will try to do the same!

[NamburiSrinath]

Here's some additional information if it helps, but I believe you would already know this, but this is there the code breaks

logits variable is --

tensor([[[ 5.1888e-02,  1.1745e+00, -2.8043e+00,  ..., -6.1696e+00,                                       
          -7.5409e+00, -8.9564e-01],                                                                      
         [ 4.1800e-01,  6.7916e-01, -2.7752e+00,  ..., -5.1078e+00,                                       
          -6.5594e+00, -9.9971e-01],                                                                      
         [ 3.7734e+00,  3.5277e+00,  1.8849e+00,  ..., -3.0128e+00,                                       
          -3.5799e+00,  3.5103e+00],                                                                      
         ...,                                                                                             
         [ 4.0497e-01,  2.9791e-01, -2.4308e+00,  ..., -3.1944e+00,                                       
          -2.9315e+00,  8.2657e-01],                                                                      
         [ 1.8392e+00,  2.6660e+00, -9.3549e-01,  ..., -6.4520e+00,                                       
          -6.0360e+00,  4.5731e-03],                                                                      
         [ 4.2277e+00,  3.5396e+00, -9.0947e-01,  ..., -5.9734e+00,                                       
          -6.8100e+00,  2.2497e+00]]], device='cuda:0') 

The labels and answers are [], [] after it's passed to slice_left() in evaluate_utils.py. I believe there's an issue with data-preprocessing?

[pengzju]

same as #43

[zxlzr]

Hi, have you solved your issue?

[NamburiSrinath]

Sorry about the delay, yes the issue has been resolved by adding this line

hparams.max_length = 100